Sep 18 19:38:11.256302 ip-10-0-142-19 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:38:11.581150 ip-10-0-142-19 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:11.581150 ip-10-0-142-19 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:38:11.581150 ip-10-0-142-19 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:11.581150 ip-10-0-142-19 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:38:11.581150 ip-10-0-142-19 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:11.581906 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.581799    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:38:11.584751 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584724    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:11.584751 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584744    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:11.584751 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584749    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:11.584751 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584753    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:11.584751 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584758    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:11.584751 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584762    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584766    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584774    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584780    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584785    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584789    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584794    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584799    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584803    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584807    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584813    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584817    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584822    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584826    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584830    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584834    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584838    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584842    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584846    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:11.585118 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584852    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584858    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584863    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584867    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584871    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584875    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584880    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584884    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584889    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584893    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584897    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584901    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584906    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584912    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584916    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584940    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584944    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584949    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584953    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:11.585886 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584967    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584971    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584975    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584980    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584984    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584990    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584994    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.584998    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585003    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585007    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585011    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585015    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585020    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585024    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585028    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585032    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585036    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585040    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585044    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585048    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:11.586705 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585052    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585057    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585061    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585065    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585069    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585073    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585077    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585081    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585088    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585092    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585096    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585100    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585105    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585109    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585113    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585117    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585121    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585125    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585131    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585135    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:11.587546 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585139    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585143    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585147    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585151    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585156    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585160    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585164    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585169    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585173    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585177    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585181    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585185    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585189    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585193    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585200    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.585206    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586426    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586436    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586441    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586445    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:11.588457 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586455    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586460    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586465    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586469    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586473    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586477    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586482    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586486    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586490    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586494    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586499    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586518    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586524    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586528    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586532    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586536    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586541    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586545    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586549    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586553    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:11.589302 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586557    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586561    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586565    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586569    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586574    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586578    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586585    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586592    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586597    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586602    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586606    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586611    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586617    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586622    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586626    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586631    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586635    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586639    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586643    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586648    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:11.590082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586652    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586656    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586660    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586665    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586677    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586682    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586686    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586690    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586694    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586706    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586711    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586715    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586719    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586723    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586727    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586731    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586735    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586739    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586744    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:11.590667 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586748    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586752    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586757    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586761    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586765    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586771    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586776    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586780    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586784    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586791    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586797    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586802    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586807    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586813    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586817    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586821    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586826    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586831    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586844    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586849    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:11.591186 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586854    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586858    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586862    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586867    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586871    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586876    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586880    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586885    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586889    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586895    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586900    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586904    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586908    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586913    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586917    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.586941    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587723    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587735    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587740    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:11.591884 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587745    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587751    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587756    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587761    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587765    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587770    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587774    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587778    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587782    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587786    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587791    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587796    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587801    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587805    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587809    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587813    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587817    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587821    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587825    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587832    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:11.592383 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587838    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587843    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587848    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587852    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587856    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587860    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587864    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587869    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587873    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587878    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587886    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587892    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587896    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587900    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587905    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587909    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587913    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587936    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:11.592941 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587944    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587950    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587954    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587958    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587963    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587966    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587972    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587976    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587980    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587984    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587990    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587994    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.587998    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588002    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588009    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588013    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588018    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588022    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588026    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588030    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:11.593448 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588035    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588039    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588043    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588048    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588053    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588058    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588062    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588067    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588071    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588076    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588080    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588084    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588088    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588092    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588096    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588100    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588104    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588108    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588112    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588117    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:11.593983 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588121    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588125    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588130    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588135    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588139    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588143    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588150    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588154    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588159    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588163    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588167    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588171    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588176    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588180    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588184    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588188    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588194    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.588198    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588311    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588362    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588372    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:38:11.594498 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588379    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588386    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588392    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588399    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588406    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588411    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588416    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588422    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588428    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588433    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588438    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588444    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588449    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588454    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588459    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588466    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588470    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588477    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588483    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588488    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588495    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588500    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588506    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:38:11.595038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588511    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588516    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588521    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588526    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588532    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588537    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588544    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588549    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588554    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588558    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588563    2560 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588568    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588575    2560 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588580    2560 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588585    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588590    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588595    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588601    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588605    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588610    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588615    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588621    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588626    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588631    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588636    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:38:11.595590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588640    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588647    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588652    2560 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588666    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588671    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588676    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588681    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588686    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588691    2560 flags.go:64] FLAG: --help="false"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588696    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-142-19.ec2.internal"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588701    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588706    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588712    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588717    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588723    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588728    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588733    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588738    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588743    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588748    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588753    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588758    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588763    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588768    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:38:11.596291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588773    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588778    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588782    2560 flags.go:64] FLAG: --lock-file=""
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588787    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588791    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588797    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588807    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588811    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588816    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588823    2560 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588828    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588833    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588838    2560 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588843    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588851    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588856    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588865    2560 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588870    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588875    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588879    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588884    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588889    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588894    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588899    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588911    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:38:11.596899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588916    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588939    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588944    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588949    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588966    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588971    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588976    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588980    2560 flags.go:64] FLAG: --port="10250"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588985    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588991    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0edbf2bc8857f4883"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.588996    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589001    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589007    2560 flags.go:64] FLAG: --register-node="true"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589013    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589019    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589024    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589031    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589036    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589042    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589048    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589053    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589058    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589062    2560 flags.go:64] FLAG: --runonce="false"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589067    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589072    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:38:11.597815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589077    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589082    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589087    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589092    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589097    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589102    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589107    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589112    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589116    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589121    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589126    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589131    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589135    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589145    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589149    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589154    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589162    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589167    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589172    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589176    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589181    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589187    2560 flags.go:64] FLAG: --v="2"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589194    2560 flags.go:64] FLAG: --version="false"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589203    2560 flags.go:64] FLAG: --vmodule=""
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589209    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:38:11.598645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.589214    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589403    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589410    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589415    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589420    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589425    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589431    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589435    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589439    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589445    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589450    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589454    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589458    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589463    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589468    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589472    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589476    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589481    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589485    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:11.599282 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589489    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589495    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589502    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589508    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589512    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589516    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589521    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589525    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589529    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589533    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589539    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589545    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589549    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589553    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589558    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589562    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589566    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589570    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589575    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589579    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:11.599755 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589583    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589588    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589595    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589600    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589604    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589608    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589612    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589616    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589620    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589625    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589629    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589634    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589638    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589642    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589647    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589651    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589655    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589659    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589664    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589668    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:11.600270 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589672    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589677    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589681    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589688    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589692    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589697    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589701    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589705    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589709    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589713    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589717    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589721    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589726    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589730    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589736    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589740    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589744    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589748    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589755    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:11.600848 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589761    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589767    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589772    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589776    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589780    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589784    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589789    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589793    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589797    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589801    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589806    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589810    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589814    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589818    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589822    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589827    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589835    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589848    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589853    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:11.601340 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589858    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589863    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.589867    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.590264    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.598601    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.598622    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.598638    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.598644    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598703    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598708    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598712    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598715    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598718    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598721    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:11.601811 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598724    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598729    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598733    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598737    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598740    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598743    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598745    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598748    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598751    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598754    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598757    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598759    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598762    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598766    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598768    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598771    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598773    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598776    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598779    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:11.602222 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598782    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598784    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598787    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598790    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598793    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598796    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598799    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598802    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598805    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598808    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598811    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598813    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598816    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598819    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598821    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598824    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598827    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598829    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598832    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598835    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:11.602694 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598838    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598841    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598844    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598847    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598849    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598852    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598854    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598857    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598860    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598862    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598865    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598867    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598870    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598873    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598875    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598878    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598882    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598885    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598888    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598890    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:11.603237 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598893    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598896    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598899    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598901    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598904    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598907    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598909    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598912    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598915    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598918    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598938    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598942    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598947    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598949    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598953    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598956    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598960    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598962    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598965    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:11.603733 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598969    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598973    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598977    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598980    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598982    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598985    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598988    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598991    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598994    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.598997    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599000    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599004    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599008    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599011    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:11.604218 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599014    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.599019    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599154    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599160    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599163    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599166    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599170    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599174    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599178    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599181    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599184    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599187    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599190    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599194    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:11.604567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599198    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599201    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599204    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599206    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599209    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599212    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599215    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599217    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599220    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599222    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599225    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599228    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599231    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599234    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599237    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599240    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599244    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599247    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599251    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599254    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:11.604999 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599257    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599260    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599263    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599265    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599268    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599271    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599273    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599276    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599279    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599281    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599284    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599287    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599290    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599294    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599297    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599299    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599302    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599305    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599308    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599311    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:11.605500 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599313    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599316    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599319    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599321    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599324    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599327    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599330    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599333    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599336    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599338    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599341    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599344    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599347    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599349    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599352    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599355    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599357    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599360    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599363    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599365    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:11.606021 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599368    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599370    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599373    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599376    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599379    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599381    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599385    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599387    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599390    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599393    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599395    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599398    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599400    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599403    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599406    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599409    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599411    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599414    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599417    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599420    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:11.606504 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599422    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599425    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599428    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599430    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599433    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599435    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:11.599438    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.599443    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.599578    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.602662    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.603625    2560 server.go:1019] "Starting client certificate rotation"
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.603706    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.604245    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:11.607010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.605760    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:11.608514 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.608497    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:11.618296 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.618268    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:38:11.620115 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.620095    2560 log.go:25] "Validated CRI v1 image API"
Sep 18 19:38:11.621688 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.621668    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:38:11.623883 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.623856    2560 fs.go:135] Filesystem UUIDs: map[5d0386e1-e662-4407-ba06-0b8cd0e4a467:/dev/nvme0n1p3 66bbd2ac-08b4-483e-bd4f-01f356df3f47:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 18 19:38:11.623964 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.623884    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:38:11.630601 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.630452    2560 manager.go:216] Machine: {Timestamp:2026-09-18 19:38:11.629161268 +0000 UTC m=+0.271450417 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099326 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c452aecb0ec114fcc434d48618ed7 SystemUUID:ec2c452a-ecb0-ec11-4fcc-434d48618ed7 BootID:70ad3f1e-8d84-40ba-8216-75cd74b6566f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ca:a3:49:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ca:a3:49:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:46:9d:bf:22:20 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:38:11.630601 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.630593    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:38:11.630767 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.630721    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:38:11.632728 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.632704    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:11.633222 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.633191    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:38:11.633416 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.633228    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-19.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:38:11.634375 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.634361    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:38:11.634409 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.634378    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:38:11.634409 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.634387    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:11.634964 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.634912    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:11.635076 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.635065    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:38:11.639197 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639171    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:11.639331 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639321    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:38:11.639369 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639356    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:38:11.639401 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639372    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:38:11.639401 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639388    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:38:11.639401 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639397    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:38:11.639489 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639407    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:38:11.639840 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639808    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:11.639901 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.639850    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:11.641785 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.641757    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:38:11.644064 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.644046    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:38:11.644150 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.644079    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:38:11.645076 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645065    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:38:11.645119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645081    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:38:11.645119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645087    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:38:11.645119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645092    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:38:11.645119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645100    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:38:11.645119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645106    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:38:11.645119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645119    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:38:11.645283 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645125    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:38:11.645283 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645132    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:38:11.645283 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645138    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:38:11.645283 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645147    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:38:11.645283 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645152    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:38:11.645859 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645848    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:38:11.645859 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.645857    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:38:11.648772 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.648744    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:38:11.648772 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.648749    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:38:11.648906 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.648755    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:38:11.649700 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.649686    2560 server.go:1268] "Started kubelet"
Sep 18 19:38:11.649800 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.649772    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:38:11.649850 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.649773    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:38:11.649850 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.649843    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:38:11.650708 ip-10-0-142-19 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:38:11.651195 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.651174    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:38:11.653174 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.653159    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:38:11.654956 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.654183    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-19.ec2.internal.18d68180b0e088eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-19.ec2.internal,UID:ip-10-0-142-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-19.ec2.internal,},FirstTimestamp:2026-09-18 19:38:11.649652971 +0000 UTC m=+0.291942119,LastTimestamp:2026-09-18 19:38:11.649652971 +0000 UTC m=+0.291942119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-19.ec2.internal,}"
Sep 18 19:38:11.655949 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.655901    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:38:11.658292 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.658272    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:11.658703 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.658686    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:38:11.658801 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.658781    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:11.659320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.659237    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:38:11.659320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.659245    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:38:11.659320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.659252    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:38:11.659541 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.659360    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:38:11.659541 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.659369    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:38:11.659541 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.659489    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-699ml"
Sep 18 19:38:11.659773 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.659755    2560 factory.go:55] Registering systemd factory
Sep 18 19:38:11.659773 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.659774    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:38:11.660036 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.660020    2560 factory.go:153] Registering CRI-O factory
Sep 18 19:38:11.660036 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.660035    2560 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:38:11.660150 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.660083    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:38:11.660150 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.660097    2560 factory.go:103] Registering Raw factory
Sep 18 19:38:11.660150 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.660108    2560 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:38:11.660546 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.660535    2560 manager.go:349] Starting recovery of all containers
Sep 18 19:38:11.660912 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.660865    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:38:11.661069 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.661046    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:38:11.666332 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.666290    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-699ml"
Sep 18 19:38:11.671471 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.671451    2560 manager.go:354] Recovery completed
Sep 18 19:38:11.676159 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.676139    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:11.679819 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.678695    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:11.679819 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.678728    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:11.679819 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.678739    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:11.679819 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.679349    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:38:11.679819 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.679359    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:38:11.679819 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.679359    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:11.679819 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.679375    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:11.680443 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.680425    2560 policy_none.go:49] "None policy: Start"
Sep 18 19:38:11.680522 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.680451    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:38:11.680522 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.680462    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:38:11.680522 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.680430    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:11.680789 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.680538    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:11.680789 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.680548    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:11.681630 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.681615    2560 policy_none.go:47] "Start"
Sep 18 19:38:11.713764 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.713743    2560 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.713797    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.713811    2560 server.go:87] "Starting device plugin registration server"
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.714324    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.714340    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.714458    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.714560    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.714570    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.715910    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:38:11.727673 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.715999    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:11.814740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.814702    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:11.815773 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.815753    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:11.815894 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.815790    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:11.815894 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.815805    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:11.815894 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.815839    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-19.ec2.internal"
Sep 18 19:38:11.823612 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.823561    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-19.ec2.internal"
Sep 18 19:38:11.823718 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.823624    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-19.ec2.internal\": node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:11.839049 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.838947    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:11.880319 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.880282    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:38:11.881477 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.881462    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:38:11.881576 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.881518    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:38:11.881576 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.881544    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:38:11.881722 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.881645    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:38:11.883845 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.883826    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:11.939592 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:11.939547    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:11.982008 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.981966    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal"]
Sep 18 19:38:11.982101 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.982060    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:11.983602 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.983587    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:11.983671 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.983614    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:11.983671 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.983626    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:11.984798 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.984787    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:11.984915 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.984902    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:11.984973 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.984952    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:11.985578 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.985567    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:11.985622 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.985595    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:11.985622 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.985605    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:11.985683 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.985567    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:11.985683 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.985672    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:11.985736 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.985685    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:11.986684 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.986663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:11.986738 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.986688    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:11.987491 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.987477    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:11.987565 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.987499    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:11.987565 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:11.987509    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:12.008661 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.008637    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-19.ec2.internal\" not found" node="ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.012522 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.012506    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-19.ec2.internal\" not found" node="ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.040467 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.040438    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:12.061141 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.061094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3463c6b633f1c9a406c989da779f6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal\" (UID: \"a3463c6b633f1c9a406c989da779f6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.061282 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.061159    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3463c6b633f1c9a406c989da779f6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal\" (UID: \"a3463c6b633f1c9a406c989da779f6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.061282 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.061185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/580e69a6077123655fe2b32c49b4aa69-config\") pod \"kube-apiserver-proxy-ip-10-0-142-19.ec2.internal\" (UID: \"580e69a6077123655fe2b32c49b4aa69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.140826 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.140743    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:12.162175 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.162145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3463c6b633f1c9a406c989da779f6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal\" (UID: \"a3463c6b633f1c9a406c989da779f6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.162278 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.162181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3463c6b633f1c9a406c989da779f6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal\" (UID: \"a3463c6b633f1c9a406c989da779f6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.162278 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.162198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/580e69a6077123655fe2b32c49b4aa69-config\") pod \"kube-apiserver-proxy-ip-10-0-142-19.ec2.internal\" (UID: \"580e69a6077123655fe2b32c49b4aa69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.162278 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.162249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3463c6b633f1c9a406c989da779f6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal\" (UID: \"a3463c6b633f1c9a406c989da779f6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.162399 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.162306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3463c6b633f1c9a406c989da779f6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal\" (UID: \"a3463c6b633f1c9a406c989da779f6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.162399 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.162336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/580e69a6077123655fe2b32c49b4aa69-config\") pod \"kube-apiserver-proxy-ip-10-0-142-19.ec2.internal\" (UID: \"580e69a6077123655fe2b32c49b4aa69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.241342 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.241305    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:12.312817 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.312777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.315809 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.315788    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.342338 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.342307    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:12.442916 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.442822    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:12.543388 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.543358    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:12.603874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.603842    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:38:12.604297 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.604010    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:12.644380 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.644344    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:12.666278 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.666255    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:12.671310 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.671282    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:33:12 +0000 UTC" deadline="2028-04-04 01:40:28.709945931 +0000 UTC"
Sep 18 19:38:12.671364 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.671311    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13518h2m16.038638221s"
Sep 18 19:38:12.744711 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:12.744630    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-19.ec2.internal\" not found"
Sep 18 19:38:12.807936 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.807885    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:12.859565 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.859533    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.873330 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.873295    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:12.875006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.874979    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal"
Sep 18 19:38:12.887325 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.887301    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:12.928308 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:12.928275    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3463c6b633f1c9a406c989da779f6b4.slice/crio-367c35deefec9c1cfe2eac6da799640c6a09b87dd64a4af3d9804fba05d1bd84 WatchSource:0}: Error finding container 367c35deefec9c1cfe2eac6da799640c6a09b87dd64a4af3d9804fba05d1bd84: Status 404 returned error can't find the container with id 367c35deefec9c1cfe2eac6da799640c6a09b87dd64a4af3d9804fba05d1bd84
Sep 18 19:38:12.932993 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:12.932972    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:38:13.025416 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:13.025372    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580e69a6077123655fe2b32c49b4aa69.slice/crio-7ac6e27e3c57f039a438e2712d2aa1602200d4fba794c20cec9711fe1305b60e WatchSource:0}: Error finding container 7ac6e27e3c57f039a438e2712d2aa1602200d4fba794c20cec9711fe1305b60e: Status 404 returned error can't find the container with id 7ac6e27e3c57f039a438e2712d2aa1602200d4fba794c20cec9711fe1305b60e
Sep 18 19:38:13.087210 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.087176    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:13.562782 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.562749    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:13.641112 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.641079    2560 apiserver.go:52] "Watching apiserver"
Sep 18 19:38:13.647831 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.647799    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:38:13.650010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.649870    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6v4ch","openshift-network-diagnostics/network-check-target-qvw9h","openshift-ovn-kubernetes/ovnkube-node-m748s","kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr","openshift-cluster-node-tuning-operator/tuned-ddpkm","openshift-dns/node-resolver-q6gg2","openshift-image-registry/node-ca-7qjmw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal","openshift-network-operator/iptables-alerter-2klft","kube-system/global-pull-secret-syncer-d579p","kube-system/konnectivity-agent-2crvg","openshift-multus/multus-additional-cni-plugins-zjk6b","openshift-multus/multus-gsd9d"]
Sep 18 19:38:13.652620 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.652590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.655291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.655265    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.656613 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.656438    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:13.656739 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.656655    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:38:13.656739 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.656690    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:13.656963 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.656939    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lkkh6\""
Sep 18 19:38:13.657465 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.657447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.658503 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.658257    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:38:13.658503 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.658333    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:38:13.658775 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.658395    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:38:13.659522 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.658475    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:38:13.659522 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.658526    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4v2kz\""
Sep 18 19:38:13.659522 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.659005    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:13.659522 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.658559    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:38:13.660680 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.659914    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:38:13.660680 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.660071    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:38:13.660680 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.660144    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:38:13.660680 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.660275    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:38:13.660680 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.660297    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.660680 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.660340    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:38:13.660680 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.660351    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-njf2c\""
Sep 18 19:38:13.662210 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.662029    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:38:13.662670 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.662344    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.662670 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.662632    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f79x2\""
Sep 18 19:38:13.662811 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.662680    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:38:13.664634 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.664614    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:38:13.664763 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.664746    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:38:13.666374 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.666205    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wpglg\""
Sep 18 19:38:13.666374 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.665019    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:38:13.669315 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.666897    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.669770 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.669747    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.669884 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.669788    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:13.670141 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.670121    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8wxc8\""
Sep 18 19:38:13.670235 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.670203    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:13.670935 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.670899    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-system-cni-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.671021 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.670957    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-cnibin\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.671021 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.670992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-socket-dir-parent\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.671124 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.671176 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-host-slash\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.671606 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671579    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpx5p\" (UniqueName: \"kubernetes.io/projected/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-kube-api-access-hpx5p\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.671718 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671611    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcjg\" (UniqueName: \"kubernetes.io/projected/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-kube-api-access-wzcjg\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.671718 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-cni-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.671718 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-k8s-cni-cncf-io\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.671718 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-etc-kubernetes\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.671909 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkvs\" (UniqueName: \"kubernetes.io/projected/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-kube-api-access-xjkvs\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.671909 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.671909 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-node-log\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.671909 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05f4058e-a32c-473e-b179-158080388120-ovn-node-metrics-cert\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.671909 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671855    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-cni-binary-copy\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.672188 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-netns\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.672188 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.671967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-hostroot\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.672188 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-daemon-config\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.672188 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.672381 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672210    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n56mj\""
Sep 18 19:38:13.672381 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-systemd-units\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672381 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-run-netns\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672381 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-ovn\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672381 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-kubelet\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672381 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-var-lib-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672381 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-env-overrides\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-cni-bin\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-iptables-alerter-script\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-slash\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-systemd\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-log-socket\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4pn7\" (UniqueName: \"kubernetes.io/projected/05f4058e-a32c-473e-b179-158080388120-kube-api-access-s4pn7\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cnibin\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.672708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-etc-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672739    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672755    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-cni-bin\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-kubelet\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-conf-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-system-cni-dir\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-run-ovn-kubernetes\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-ovnkube-config\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-ovnkube-script-lib\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-cni-multus\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-multus-certs\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.672993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-cni-netd\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.673218 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.673058    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:38:13.673798 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.673228    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:38:13.673798 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.673270    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:13.673798 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.673558    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.675829 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.675808    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-75q56\""
Sep 18 19:38:13.676582 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.676130    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:38:13.676582 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.676195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:13.676582 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.676263    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:13.676582 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.676430    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:38:13.678756 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.678737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:13.678849 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.678818    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:13.681245 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.681225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.681335 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.681305    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:13.683632 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.683611    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:13.686035 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.686006    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8vdd7\""
Sep 18 19:38:13.686177 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.686148    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:38:13.686177 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.686172    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:38:13.761514 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.761488    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:38:13.773377 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.773377 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4555a60-1138-42ec-8931-6b4f9091edc3-host\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.773630 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-run\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.773630 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw5px\" (UniqueName: \"kubernetes.io/projected/7bce6468-88f5-4663-988a-a81cfbb2a9aa-kube-api-access-qw5px\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.773630 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-var-lib-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.773630 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-cni-bin\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.773630 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-var-lib-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.773630 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-iptables-alerter-script\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-slash\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-cni-bin\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7tl\" (UniqueName: \"kubernetes.io/projected/6c50101d-94f2-4e37-804a-2275d74515ba-kube-api-access-tz7tl\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8cbf7a2-e0da-417e-8353-9397e53a62d0-tmp-dir\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-slash\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-sys\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-etc-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.773951 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-etc-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-openvswitch\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.773947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-ovnkube-config\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-ovnkube-script-lib\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-multus-certs\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-host\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-multus-certs\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-system-cni-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-cnibin\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-socket-dir-parent\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774277    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1-agent-certs\") pod \"konnectivity-agent-2crvg\" (UID: \"2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1\") " pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-host-slash\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-k8s-cni-cncf-io\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-system-cni-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkvs\" (UniqueName: \"kubernetes.io/projected/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-kube-api-access-xjkvs\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05f4058e-a32c-473e-b179-158080388120-ovn-node-metrics-cert\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-cni-binary-copy\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.774492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-iptables-alerter-script\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-hostroot\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-daemon-config\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-systemd-units\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-run-netns\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-ovn\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774783    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-ovnkube-script-lib\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-registration-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-ovnkube-config\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8cbf7a2-e0da-417e-8353-9397e53a62d0-hosts-file\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-host-slash\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774899    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-kubelet\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-systemd-units\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-socket-dir-parent\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-k8s-cni-cncf-io\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-env-overrides\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.775337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.774975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-kubelet\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.775014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-socket-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.775051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-hostroot\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.775846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.775896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6972273-7829-45b0-9158-bfada0b133e1-kubelet-config\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.775982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-var-lib-kubelet\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-tuned\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776016    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wvtsc"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-systemd\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776065    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-log-socket\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.776127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-cni-binary-copy\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4pn7\" (UniqueName: \"kubernetes.io/projected/05f4058e-a32c-473e-b179-158080388120-kube-api-access-s4pn7\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-kubelet\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-conf-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tmt\" (UniqueName: \"kubernetes.io/projected/e4555a60-1138-42ec-8931-6b4f9091edc3-kube-api-access-59tmt\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-daemon-config\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776272    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cnibin\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-log-socket\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-cni-bin\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05f4058e-a32c-473e-b179-158080388120-env-overrides\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776550    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-cnibin\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776649    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-systemd\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.776674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-kubelet\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-conf-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776763    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-run-netns\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cnibin\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-run-ovn\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-cni-bin\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776895    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.776979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-modprobe-d\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysctl-d\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysctl-conf\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-system-cni-dir\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-run-ovn-kubernetes\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.777187 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777193    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-cni-multus\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777221    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysconfig\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-kubernetes\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-cni-netd\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4555a60-1138-42ec-8931-6b4f9091edc3-serviceca\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-device-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhljb\" (UniqueName: \"kubernetes.io/projected/d583d577-4430-4736-9cff-606fe6ee92e9-kube-api-access-xhljb\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1-konnectivity-ca\") pod \"konnectivity-agent-2crvg\" (UID: \"2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1\") " pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777458    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-lib-modules\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777515    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpx5p\" (UniqueName: \"kubernetes.io/projected/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-kube-api-access-hpx5p\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcjg\" (UniqueName: \"kubernetes.io/projected/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-kube-api-access-wzcjg\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-cni-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-etc-kubernetes\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6972273-7829-45b0-9158-bfada0b133e1-dbus\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.777835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bce6468-88f5-4663-988a-a81cfbb2a9aa-tmp\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-node-log\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-netns\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-sys-fs\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qd7q\" (UniqueName: \"kubernetes.io/projected/d8cbf7a2-e0da-417e-8353-9397e53a62d0-kube-api-access-9qd7q\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-systemd\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.777953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-system-cni-dir\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.778014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-run-ovn-kubernetes\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.778063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-var-lib-cni-multus\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.778131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-cni-netd\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.778518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.778251    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.779012 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.778759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-multus-cni-dir\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.779012 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.778816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-etc-kubernetes\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.780075 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.779221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-host-run-netns\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.780075 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.779283    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05f4058e-a32c-473e-b179-158080388120-node-log\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.780075 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.779355    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.780906 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.780881    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05f4058e-a32c-473e-b179-158080388120-ovn-node-metrics-cert\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.784343 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.784320    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wvtsc"
Sep 18 19:38:13.786095 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.786061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkvs\" (UniqueName: \"kubernetes.io/projected/79834c8f-5e1f-4ddf-a2fc-0b393da1343b-kube-api-access-xjkvs\") pod \"multus-gsd9d\" (UID: \"79834c8f-5e1f-4ddf-a2fc-0b393da1343b\") " pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:13.791466 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.791441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4pn7\" (UniqueName: \"kubernetes.io/projected/05f4058e-a32c-473e-b179-158080388120-kube-api-access-s4pn7\") pod \"ovnkube-node-m748s\" (UID: \"05f4058e-a32c-473e-b179-158080388120\") " pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.794785 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.794736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpx5p\" (UniqueName: \"kubernetes.io/projected/4f33dd29-44ca-43b1-bfb0-0a38b154ee58-kube-api-access-hpx5p\") pod \"iptables-alerter-2klft\" (UID: \"4f33dd29-44ca-43b1-bfb0-0a38b154ee58\") " pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.795873 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.795842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcjg\" (UniqueName: \"kubernetes.io/projected/5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6-kube-api-access-wzcjg\") pod \"multus-additional-cni-plugins-zjk6b\" (UID: \"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6\") " pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.878473 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4555a60-1138-42ec-8931-6b4f9091edc3-serviceca\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.878473 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-device-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.878473 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhljb\" (UniqueName: \"kubernetes.io/projected/d583d577-4430-4736-9cff-606fe6ee92e9-kube-api-access-xhljb\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1-konnectivity-ca\") pod \"konnectivity-agent-2crvg\" (UID: \"2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1\") " pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-lib-modules\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6972273-7829-45b0-9158-bfada0b133e1-dbus\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bce6468-88f5-4663-988a-a81cfbb2a9aa-tmp\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-sys-fs\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qd7q\" (UniqueName: \"kubernetes.io/projected/d8cbf7a2-e0da-417e-8353-9397e53a62d0-kube-api-access-9qd7q\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878716    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-systemd\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-lib-modules\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.878740 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4555a60-1138-42ec-8931-6b4f9091edc3-host\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-run\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4555a60-1138-42ec-8931-6b4f9091edc3-host\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6972273-7829-45b0-9158-bfada0b133e1-dbus\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw5px\" (UniqueName: \"kubernetes.io/projected/7bce6468-88f5-4663-988a-a81cfbb2a9aa-kube-api-access-qw5px\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7tl\" (UniqueName: \"kubernetes.io/projected/6c50101d-94f2-4e37-804a-2275d74515ba-kube-api-access-tz7tl\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8cbf7a2-e0da-417e-8353-9397e53a62d0-tmp-dir\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-sys\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-host\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878899    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-sys-fs\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1-agent-certs\") pod \"konnectivity-agent-2crvg\" (UID: \"2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1\") " pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4555a60-1138-42ec-8931-6b4f9091edc3-serviceca\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-systemd\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.878989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-registration-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-device-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879049    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-sys\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8cbf7a2-e0da-417e-8353-9397e53a62d0-hosts-file\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.879272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-socket-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6972273-7829-45b0-9158-bfada0b133e1-kubelet-config\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-var-lib-kubelet\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-tuned\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tmt\" (UniqueName: \"kubernetes.io/projected/e4555a60-1138-42ec-8931-6b4f9091edc3-kube-api-access-59tmt\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-run\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8cbf7a2-e0da-417e-8353-9397e53a62d0-tmp-dir\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-socket-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879381    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-modprobe-d\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.879393    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysctl-d\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-registration-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c50101d-94f2-4e37-804a-2275d74515ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysctl-conf\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879467    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-host\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.879484    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret podName:d6972273-7829-45b0-9158-bfada0b133e1 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:14.379440034 +0000 UTC m=+3.021729206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret") pod "global-pull-secret-syncer-d579p" (UID: "d6972273-7829-45b0-9158-bfada0b133e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysctl-conf\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879606    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8cbf7a2-e0da-417e-8353-9397e53a62d0-hosts-file\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysconfig\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879674    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6972273-7829-45b0-9158-bfada0b133e1-kubelet-config\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-kubernetes\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-kubernetes\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysconfig\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879766    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-modprobe-d\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-var-lib-kubelet\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.879810    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.879909    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-sysctl-d\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.879917    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs podName:d583d577-4430-4736-9cff-606fe6ee92e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:14.379900407 +0000 UTC m=+3.022189563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs") pod "network-metrics-daemon-6v4ch" (UID: "d583d577-4430-4736-9cff-606fe6ee92e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:13.880874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.880340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1-konnectivity-ca\") pod \"konnectivity-agent-2crvg\" (UID: \"2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1\") " pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:13.883847 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.883782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bce6468-88f5-4663-988a-a81cfbb2a9aa-etc-tuned\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.883847 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.883810    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bce6468-88f5-4663-988a-a81cfbb2a9aa-tmp\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.884049 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.883989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1-agent-certs\") pod \"konnectivity-agent-2crvg\" (UID: \"2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1\") " pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:13.886833 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.886319    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:13.886833 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.886347    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:13.886833 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.886362    2560 projected.go:196] Error preparing data for projected volume kube-api-access-nzfx6 for pod openshift-network-diagnostics/network-check-target-qvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:13.886833 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:13.886430    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6 podName:e6c68d39-1394-4fea-969f-422a6817eb6b nodeName:}" failed. No retries permitted until 2026-09-18 19:38:14.386410297 +0000 UTC m=+3.028699435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nzfx6" (UniqueName: "kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6") pod "network-check-target-qvw9h" (UID: "e6c68d39-1394-4fea-969f-422a6817eb6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:13.887684 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.887617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qd7q\" (UniqueName: \"kubernetes.io/projected/d8cbf7a2-e0da-417e-8353-9397e53a62d0-kube-api-access-9qd7q\") pod \"node-resolver-q6gg2\" (UID: \"d8cbf7a2-e0da-417e-8353-9397e53a62d0\") " pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:13.887895 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.887869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhljb\" (UniqueName: \"kubernetes.io/projected/d583d577-4430-4736-9cff-606fe6ee92e9-kube-api-access-xhljb\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:13.888401 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.888352    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tmt\" (UniqueName: \"kubernetes.io/projected/e4555a60-1138-42ec-8931-6b4f9091edc3-kube-api-access-59tmt\") pod \"node-ca-7qjmw\" (UID: \"e4555a60-1138-42ec-8931-6b4f9091edc3\") " pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:13.890208 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.890186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw5px\" (UniqueName: \"kubernetes.io/projected/7bce6468-88f5-4663-988a-a81cfbb2a9aa-kube-api-access-qw5px\") pod \"tuned-ddpkm\" (UID: \"7bce6468-88f5-4663-988a-a81cfbb2a9aa\") " pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:13.890286 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.890235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal" event={"ID":"580e69a6077123655fe2b32c49b4aa69","Type":"ContainerStarted","Data":"7ac6e27e3c57f039a438e2712d2aa1602200d4fba794c20cec9711fe1305b60e"}
Sep 18 19:38:13.890412 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.890389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7tl\" (UniqueName: \"kubernetes.io/projected/6c50101d-94f2-4e37-804a-2275d74515ba-kube-api-access-tz7tl\") pod \"aws-ebs-csi-driver-node-tbzvr\" (UID: \"6c50101d-94f2-4e37-804a-2275d74515ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:13.892286 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.892213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal" event={"ID":"a3463c6b633f1c9a406c989da779f6b4","Type":"ContainerStarted","Data":"367c35deefec9c1cfe2eac6da799640c6a09b87dd64a4af3d9804fba05d1bd84"}
Sep 18 19:38:13.969875 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.969840    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2klft"
Sep 18 19:38:13.980275 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.980241    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjk6b"
Sep 18 19:38:13.990126 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.990095    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:13.998155 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:13.998111    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gsd9d"
Sep 18 19:38:14.004964 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:14.004934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr"
Sep 18 19:38:14.011789 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:14.011763    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ddpkm"
Sep 18 19:38:14.018635 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:14.018604    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7qjmw"
Sep 18 19:38:14.027400 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:14.027376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6gg2"
Sep 18 19:38:14.034281 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:14.034254    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:15.422746 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.422708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:15.422948 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.422762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:15.422948 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.422870    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:15.422948 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.422900    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:15.422948 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.422939    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs podName:d583d577-4430-4736-9cff-606fe6ee92e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:16.422920321 +0000 UTC m=+4.025283485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs") pod "network-metrics-daemon-6v4ch" (UID: "d583d577-4430-4736-9cff-606fe6ee92e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:15.423105 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.422959    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret podName:d6972273-7829-45b0-9158-bfada0b133e1 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:16.422949585 +0000 UTC m=+4.025312745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret") pod "global-pull-secret-syncer-d579p" (UID: "d6972273-7829-45b0-9158-bfada0b133e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:15.523412 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.523331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:15.523565 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.523542    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:15.523620 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.523570    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:15.523620 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.523583    2560 projected.go:196] Error preparing data for projected volume kube-api-access-nzfx6 for pod openshift-network-diagnostics/network-check-target-qvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:15.523702 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.523652    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6 podName:e6c68d39-1394-4fea-969f-422a6817eb6b nodeName:}" failed. No retries permitted until 2026-09-18 19:38:16.52363167 +0000 UTC m=+4.125994846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzfx6" (UniqueName: "kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6") pod "network-check-target-qvw9h" (UID: "e6c68d39-1394-4fea-969f-422a6817eb6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:15.829048 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.828971    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:14 +0000 UTC" deadline="2028-04-05 02:35:52.538446426 +0000 UTC"
Sep 18 19:38:15.829048 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.829002    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13542h57m36.709448417s"
Sep 18 19:38:15.850769 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:15.850732    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0e4816_d28a_4f5e_b22e_c0fe86bd51d6.slice/crio-5e79633c3127ea068e64f1faf1e2ae5ddc9e1dd3c029186aea42afe09a5ccd50 WatchSource:0}: Error finding container 5e79633c3127ea068e64f1faf1e2ae5ddc9e1dd3c029186aea42afe09a5ccd50: Status 404 returned error can't find the container with id 5e79633c3127ea068e64f1faf1e2ae5ddc9e1dd3c029186aea42afe09a5ccd50
Sep 18 19:38:15.856708 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:15.856500    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c50101d_94f2_4e37_804a_2275d74515ba.slice/crio-6529c4c134f46732209c635034a9a36dcf2e39658aa2cc61e5cfa8c2ce63f24b WatchSource:0}: Error finding container 6529c4c134f46732209c635034a9a36dcf2e39658aa2cc61e5cfa8c2ce63f24b: Status 404 returned error can't find the container with id 6529c4c134f46732209c635034a9a36dcf2e39658aa2cc61e5cfa8c2ce63f24b
Sep 18 19:38:15.878323 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:15.878289    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79834c8f_5e1f_4ddf_a2fc_0b393da1343b.slice/crio-168a28bd3758e39fd1012199bef819a71e386379135715343b224bc58c926b15 WatchSource:0}: Error finding container 168a28bd3758e39fd1012199bef819a71e386379135715343b224bc58c926b15: Status 404 returned error can't find the container with id 168a28bd3758e39fd1012199bef819a71e386379135715343b224bc58c926b15
Sep 18 19:38:15.879991 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:15.879610    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cbf7a2_e0da_417e_8353_9397e53a62d0.slice/crio-f6da2ad1995bb778d5c22db876be9c62e88b6791471b81fa2bbcda85e77c7fb6 WatchSource:0}: Error finding container f6da2ad1995bb778d5c22db876be9c62e88b6791471b81fa2bbcda85e77c7fb6: Status 404 returned error can't find the container with id f6da2ad1995bb778d5c22db876be9c62e88b6791471b81fa2bbcda85e77c7fb6
Sep 18 19:38:15.880165 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:15.880027    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4555a60_1138_42ec_8931_6b4f9091edc3.slice/crio-50df685a145cdf2a1f59b786c8e6d50331ba5da4fffe9d0d8fce8a6c271c3712 WatchSource:0}: Error finding container 50df685a145cdf2a1f59b786c8e6d50331ba5da4fffe9d0d8fce8a6c271c3712: Status 404 returned error can't find the container with id 50df685a145cdf2a1f59b786c8e6d50331ba5da4fffe9d0d8fce8a6c271c3712
Sep 18 19:38:15.880962 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:15.880888    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f33dd29_44ca_43b1_bfb0_0a38b154ee58.slice/crio-37175959b3351c9e643ee1ab1945d25cdbfb720db69880737f13f0e32d70c917 WatchSource:0}: Error finding container 37175959b3351c9e643ee1ab1945d25cdbfb720db69880737f13f0e32d70c917: Status 404 returned error can't find the container with id 37175959b3351c9e643ee1ab1945d25cdbfb720db69880737f13f0e32d70c917
Sep 18 19:38:15.883099 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:15.882808    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bce6468_88f5_4663_988a_a81cfbb2a9aa.slice/crio-b2a5923acc93e0edf7f22ba85462c0eb9df6983c5a5c8e3e75c2c3546f02f417 WatchSource:0}: Error finding container b2a5923acc93e0edf7f22ba85462c0eb9df6983c5a5c8e3e75c2c3546f02f417: Status 404 returned error can't find the container with id b2a5923acc93e0edf7f22ba85462c0eb9df6983c5a5c8e3e75c2c3546f02f417
Sep 18 19:38:15.921720 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.921691    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:15.921874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.921703    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:15.921919 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.921842    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:15.921964 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:15.921915    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:15.935422 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.935378    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ddpkm" event={"ID":"7bce6468-88f5-4663-988a-a81cfbb2a9aa","Type":"ContainerStarted","Data":"b2a5923acc93e0edf7f22ba85462c0eb9df6983c5a5c8e3e75c2c3546f02f417"}
Sep 18 19:38:15.936753 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.936720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2klft" event={"ID":"4f33dd29-44ca-43b1-bfb0-0a38b154ee58","Type":"ContainerStarted","Data":"37175959b3351c9e643ee1ab1945d25cdbfb720db69880737f13f0e32d70c917"}
Sep 18 19:38:15.938042 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.938018    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"ee268ec3f7a76ed4e4120e70868383f615a31f35ba9bb7020ea190fe4a0d1c79"}
Sep 18 19:38:15.939160 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.939139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2crvg" event={"ID":"2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1","Type":"ContainerStarted","Data":"bd62d4450ff0687e877d2fa92c876d99f0946fadd9cb7855efd2214d1cdd6925"}
Sep 18 19:38:15.940485 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.940458    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" event={"ID":"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6","Type":"ContainerStarted","Data":"5e79633c3127ea068e64f1faf1e2ae5ddc9e1dd3c029186aea42afe09a5ccd50"}
Sep 18 19:38:15.941745 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.941710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gsd9d" event={"ID":"79834c8f-5e1f-4ddf-a2fc-0b393da1343b","Type":"ContainerStarted","Data":"168a28bd3758e39fd1012199bef819a71e386379135715343b224bc58c926b15"}
Sep 18 19:38:15.942988 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.942957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7qjmw" event={"ID":"e4555a60-1138-42ec-8931-6b4f9091edc3","Type":"ContainerStarted","Data":"50df685a145cdf2a1f59b786c8e6d50331ba5da4fffe9d0d8fce8a6c271c3712"}
Sep 18 19:38:15.944251 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.944212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6gg2" event={"ID":"d8cbf7a2-e0da-417e-8353-9397e53a62d0","Type":"ContainerStarted","Data":"f6da2ad1995bb778d5c22db876be9c62e88b6791471b81fa2bbcda85e77c7fb6"}
Sep 18 19:38:15.945312 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:15.945291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr" event={"ID":"6c50101d-94f2-4e37-804a-2275d74515ba","Type":"ContainerStarted","Data":"6529c4c134f46732209c635034a9a36dcf2e39658aa2cc61e5cfa8c2ce63f24b"}
Sep 18 19:38:16.430708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.429829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:16.430708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.429900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:16.430708 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.430080    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:16.430708 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.430159    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret podName:d6972273-7829-45b0-9158-bfada0b133e1 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:18.430124335 +0000 UTC m=+6.032487513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret") pod "global-pull-secret-syncer-d579p" (UID: "d6972273-7829-45b0-9158-bfada0b133e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:16.430708 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.430560    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:16.430708 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.430608    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs podName:d583d577-4430-4736-9cff-606fe6ee92e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:18.43059318 +0000 UTC m=+6.032956347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs") pod "network-metrics-daemon-6v4ch" (UID: "d583d577-4430-4736-9cff-606fe6ee92e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:16.530861 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.530811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:16.531069 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.531033    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:16.531069 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.531054    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:16.531069 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.531068    2560 projected.go:196] Error preparing data for projected volume kube-api-access-nzfx6 for pod openshift-network-diagnostics/network-check-target-qvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:16.531224 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.531127    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6 podName:e6c68d39-1394-4fea-969f-422a6817eb6b nodeName:}" failed. No retries permitted until 2026-09-18 19:38:18.531109194 +0000 UTC m=+6.133472355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzfx6" (UniqueName: "kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6") pod "network-check-target-qvw9h" (UID: "e6c68d39-1394-4fea-969f-422a6817eb6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:16.829828 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.829734    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:14 +0000 UTC" deadline="2028-05-13 15:01:21.602263613 +0000 UTC"
Sep 18 19:38:16.829828 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.829778    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14467h23m4.772490817s"
Sep 18 19:38:16.925394 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.924888    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:16.925394 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:16.925025    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:16.963262 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.963211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal" event={"ID":"580e69a6077123655fe2b32c49b4aa69","Type":"ContainerStarted","Data":"28ea4e4660e3c800b7e47415f2bb628cee11bd20ff55564d541c6f3fd6699606"}
Sep 18 19:38:16.973796 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.973115    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3463c6b633f1c9a406c989da779f6b4" containerID="ce2c618735e0d4879d6a0ef51e4a1202edd7f3a41f16a13651c15f7871af53dd" exitCode=0
Sep 18 19:38:16.973796 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.973179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal" event={"ID":"a3463c6b633f1c9a406c989da779f6b4","Type":"ContainerDied","Data":"ce2c618735e0d4879d6a0ef51e4a1202edd7f3a41f16a13651c15f7871af53dd"}
Sep 18 19:38:16.979010 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:16.978667    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-19.ec2.internal" podStartSLOduration=3.978644978 podStartE2EDuration="3.978644978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:16.978073138 +0000 UTC m=+4.580436346" watchObservedRunningTime="2026-09-18 19:38:16.978644978 +0000 UTC m=+4.581008163"
Sep 18 19:38:17.922718 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:17.922030    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:17.922718 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:17.922173    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:17.922718 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:17.922568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:17.922718 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:17.922666    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:17.992431 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:17.992222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal" event={"ID":"a3463c6b633f1c9a406c989da779f6b4","Type":"ContainerStarted","Data":"c745b18aaa981d2858c60bd03faf46b2c720607b4710c7a6192ae7047af71504"}
Sep 18 19:38:18.448279 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:18.448187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:18.448279 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:18.448248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:18.448501 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.448364    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:18.448907 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.448878    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs podName:d583d577-4430-4736-9cff-606fe6ee92e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:22.448836619 +0000 UTC m=+10.051199799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs") pod "network-metrics-daemon-6v4ch" (UID: "d583d577-4430-4736-9cff-606fe6ee92e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:18.454041 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.453947    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:18.454192 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.454076    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret podName:d6972273-7829-45b0-9158-bfada0b133e1 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:22.454020122 +0000 UTC m=+10.056383299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret") pod "global-pull-secret-syncer-d579p" (UID: "d6972273-7829-45b0-9158-bfada0b133e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:18.549624 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:18.549587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:18.549831 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.549813    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:18.549942 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.549841    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:18.549942 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.549869    2560 projected.go:196] Error preparing data for projected volume kube-api-access-nzfx6 for pod openshift-network-diagnostics/network-check-target-qvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:18.549942 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.549932    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6 podName:e6c68d39-1394-4fea-969f-422a6817eb6b nodeName:}" failed. No retries permitted until 2026-09-18 19:38:22.549911295 +0000 UTC m=+10.152274459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzfx6" (UniqueName: "kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6") pod "network-check-target-qvw9h" (UID: "e6c68d39-1394-4fea-969f-422a6817eb6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:18.924805 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:18.924719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:18.925289 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:18.924934    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:19.922068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:19.922033    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:19.922275 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:19.922247    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:19.922773 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:19.922751    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:19.922968 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:19.922878    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:20.922593 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:20.922539    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:20.923077 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:20.922663    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:21.922239 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:21.922155    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:21.922477 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:21.922300    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:21.922668 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:21.922648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:21.923060 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:21.922758    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:22.484507 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:22.484468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:22.484710 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:22.484569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:22.484710 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.484627    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:22.484710 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.484690    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:22.484710 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.484702    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret podName:d6972273-7829-45b0-9158-bfada0b133e1 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.484682701 +0000 UTC m=+18.087045877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret") pod "global-pull-secret-syncer-d579p" (UID: "d6972273-7829-45b0-9158-bfada0b133e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:22.484938 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.484741    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs podName:d583d577-4430-4736-9cff-606fe6ee92e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.484725776 +0000 UTC m=+18.087088941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs") pod "network-metrics-daemon-6v4ch" (UID: "d583d577-4430-4736-9cff-606fe6ee92e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:22.585810 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:22.585758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:22.586022 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.585971    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:22.586022 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.585992    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:22.586022 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.586004    2560 projected.go:196] Error preparing data for projected volume kube-api-access-nzfx6 for pod openshift-network-diagnostics/network-check-target-qvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:22.586176 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.586069    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6 podName:e6c68d39-1394-4fea-969f-422a6817eb6b nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.586050476 +0000 UTC m=+18.188413656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzfx6" (UniqueName: "kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6") pod "network-check-target-qvw9h" (UID: "e6c68d39-1394-4fea-969f-422a6817eb6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:22.925176 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:22.924401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:22.925176 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:22.924617    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:23.922332 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:23.922037    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:23.922332 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:23.922141    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:23.922591 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:23.922402    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:23.922591 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:23.922536    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:24.921983 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:24.921940    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:24.922386 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:24.922097    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:25.922647 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:25.922610    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:25.923119 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:25.922762    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:25.923119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:25.922824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:25.923119 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:25.922959    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:26.921805 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:26.921710    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:26.921976 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:26.921861    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:27.922097 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:27.922052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:27.922572 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:27.922069    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:27.922572 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:27.922208    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:27.922572 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:27.922279    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:28.922753 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:28.922517    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:28.923208 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:28.922885    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:29.922235 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:29.922189    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:29.922235 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:29.922232    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:29.922548 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:29.922348    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:29.922548 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:29.922500    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:30.549233 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:30.549138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:30.549233 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:30.549230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:30.549736 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.549300    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:30.549736 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.549334    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:30.549736 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.549390    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret podName:d6972273-7829-45b0-9158-bfada0b133e1 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:46.549365726 +0000 UTC m=+34.151728917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret") pod "global-pull-secret-syncer-d579p" (UID: "d6972273-7829-45b0-9158-bfada0b133e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:30.549736 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.549412    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs podName:d583d577-4430-4736-9cff-606fe6ee92e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:46.549402254 +0000 UTC m=+34.151765416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs") pod "network-metrics-daemon-6v4ch" (UID: "d583d577-4430-4736-9cff-606fe6ee92e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:30.649759 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:30.649718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:30.649979 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.649901    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:30.649979 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.649923    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:30.649979 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.649937    2560 projected.go:196] Error preparing data for projected volume kube-api-access-nzfx6 for pod openshift-network-diagnostics/network-check-target-qvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:30.650130 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.650004    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6 podName:e6c68d39-1394-4fea-969f-422a6817eb6b nodeName:}" failed. No retries permitted until 2026-09-18 19:38:46.649986455 +0000 UTC m=+34.252349617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzfx6" (UniqueName: "kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6") pod "network-check-target-qvw9h" (UID: "e6c68d39-1394-4fea-969f-422a6817eb6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:30.922344 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:30.922229    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:30.922517 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:30.922356    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:31.922088 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:31.922053    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:31.922474 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:31.922054    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:31.922474 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:31.922168    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:31.922474 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:31.922254    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:32.922924 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:32.922887    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:32.923359 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:32.923000    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:33.922057 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:33.922022    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:33.922195 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:33.922020    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:33.922195 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:33.922174    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:33.922289 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:33.922215    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:34.922310 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:34.922073    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:34.923160 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:34.922380    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:35.030182 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.030145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ddpkm" event={"ID":"7bce6468-88f5-4663-988a-a81cfbb2a9aa","Type":"ContainerStarted","Data":"a2fed2f95729c4c56bf9a11354b5eade6e6dff77bc0dc0536c8b64f7fbcb493e"}
Sep 18 19:38:35.033304 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.033276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"8fba0783aefef4fc1ed719627823bb6d5a3a06e59ca3e71e72489df4a7b9b746"}
Sep 18 19:38:35.033471 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.033340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"6c93922322fa69d52ebe7a407f345ff29a9c11d69fd413896f9ba01a7e89c021"}
Sep 18 19:38:35.033471 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.033369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"f41e72cad3bf1d2a214285486a3d82c80698983fb69a6789c685b994bdef68f7"}
Sep 18 19:38:35.033471 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.033397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"d36f9b89d55274b3d75ee05d6c581ffacb7b2a3a042ed3585f70c4395e9b3c69"}
Sep 18 19:38:35.033471 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.033435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"bc0a0ad89ecdb80764c2db558d796581e971063d672298f16ae94ba2ca89dee4"}
Sep 18 19:38:35.033471 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.033466    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"888b42ca1607a0dd894701d023b9e69e4d68af9d7fdd0db0ac1f3cbc03a2b1ff"}
Sep 18 19:38:35.034730 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.034704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2crvg" event={"ID":"2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1","Type":"ContainerStarted","Data":"00f2d80f277b5e044605212e0bbcdad338c324876afd38d56c497ee7433730f6"}
Sep 18 19:38:35.036189 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.036166    2560 generic.go:356] "Generic (PLEG): container finished" podID="5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6" containerID="3ba28ff05771d8981e94125b803c0a12f47561086a768a2b9f7a0dc8639f89f8" exitCode=0
Sep 18 19:38:35.036303 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.036211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" event={"ID":"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6","Type":"ContainerDied","Data":"3ba28ff05771d8981e94125b803c0a12f47561086a768a2b9f7a0dc8639f89f8"}
Sep 18 19:38:35.037765 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.037741    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gsd9d" event={"ID":"79834c8f-5e1f-4ddf-a2fc-0b393da1343b","Type":"ContainerStarted","Data":"b88cabc65c66f1751cb57a2962632f752b32bd2eadbaf6c7fef1bc59f18b9e7b"}
Sep 18 19:38:35.039345 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.039326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7qjmw" event={"ID":"e4555a60-1138-42ec-8931-6b4f9091edc3","Type":"ContainerStarted","Data":"9290a05021f53ef93b320b34e5944f181f175ec53f4e698138d15de9c0edc464"}
Sep 18 19:38:35.040721 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.040693    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6gg2" event={"ID":"d8cbf7a2-e0da-417e-8353-9397e53a62d0","Type":"ContainerStarted","Data":"0e0c54d05a4c46c3a4b8ff32974294b747b5b29099b0e416ccdbb148b0a23323"}
Sep 18 19:38:35.042149 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.042129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr" event={"ID":"6c50101d-94f2-4e37-804a-2275d74515ba","Type":"ContainerStarted","Data":"8439b680a9a3833d7095976b79bcc6747f71e70f525dfbd553260071f539aca5"}
Sep 18 19:38:35.049943 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.049893    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-19.ec2.internal" podStartSLOduration=22.049881762 podStartE2EDuration="22.049881762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:18.007455459 +0000 UTC m=+5.609818643" watchObservedRunningTime="2026-09-18 19:38:35.049881762 +0000 UTC m=+22.652244943"
Sep 18 19:38:35.050466 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.050440    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ddpkm" podStartSLOduration=4.9777028 podStartE2EDuration="23.050434507s" totalImagesPullingTime="18.072731707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.049762301 +0000 UTC m=+22.652125483" watchObservedRunningTime="2026-09-18 19:38:35.050434507 +0000 UTC m=+22.652797689"
Sep 18 19:38:35.052911 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.052889    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:35.053403 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.053385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:35.065800 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.065755    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2crvg" podStartSLOduration=8.6651358 podStartE2EDuration="23.065741648s" totalImagesPullingTime="14.400605848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.065530228 +0000 UTC m=+22.667893411" watchObservedRunningTime="2026-09-18 19:38:35.065741648 +0000 UTC m=+22.668104876"
Sep 18 19:38:35.083179 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.083129    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gsd9d" podStartSLOduration=4.975074662 podStartE2EDuration="23.083111628s" totalImagesPullingTime="18.108036966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.083021918 +0000 UTC m=+22.685385099" watchObservedRunningTime="2026-09-18 19:38:35.083111628 +0000 UTC m=+22.685474809"
Sep 18 19:38:35.099968 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.099918    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q6gg2" podStartSLOduration=5.019643109 podStartE2EDuration="23.099900457s" totalImagesPullingTime="18.080257348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.099479758 +0000 UTC m=+22.701842939" watchObservedRunningTime="2026-09-18 19:38:35.099900457 +0000 UTC m=+22.702263641"
Sep 18 19:38:35.135604 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.135557    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7qjmw" podStartSLOduration=5.077629104 podStartE2EDuration="23.135529428s" totalImagesPullingTime="18.057900324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.134978365 +0000 UTC m=+22.737341545" watchObservedRunningTime="2026-09-18 19:38:35.135529428 +0000 UTC m=+22.737892610"
Sep 18 19:38:35.216469 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.216426    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:35.269075 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.269039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:35.331241 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.331203    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:35.433651 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.433408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:35.494932 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.494844    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:38:35.533710 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.533670    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:35.748609 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.748522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:35.768447 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.768324    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:38:35.494883664Z","UUID":"6bc9adb7-7ed5-4bdc-86fe-de87f436c058","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:38:35.770584 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.770554    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:38:35.770731 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.770595    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:38:35.922360 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.922325    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:35.922842 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:35.922452    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:35.922842 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:35.922510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:35.922842 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:35.922605    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:36.046599 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.046514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2klft" event={"ID":"4f33dd29-44ca-43b1-bfb0-0a38b154ee58","Type":"ContainerStarted","Data":"1501127a8e610084b049b4ed8c910e5a36395184ed8dbef201b2dad00c3fc6f5"}
Sep 18 19:38:36.049223 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.049145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr" event={"ID":"6c50101d-94f2-4e37-804a-2275d74515ba","Type":"ContainerStarted","Data":"a2da6db1abd9cc9abbe3ea06f327d837aaab531711ca65a331a9592088d7088b"}
Sep 18 19:38:36.049742 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.049667    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:36.050314 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.050296    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2crvg"
Sep 18 19:38:36.064802 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.064746    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2klft" podStartSLOduration=5.9937572679999995 podStartE2EDuration="24.064726203s" totalImagesPullingTime="18.070968935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:36.06425123 +0000 UTC m=+23.666614413" watchObservedRunningTime="2026-09-18 19:38:36.064726203 +0000 UTC m=+23.667089384"
Sep 18 19:38:36.196368 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.196314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:36.596597 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.596567    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:36.695191 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.695143    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:36.843545 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.843453    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:36.922449 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.922416    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:36.922936 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:36.922544    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:36.962557 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:36.962525    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:37.057907 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.057829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr" event={"ID":"6c50101d-94f2-4e37-804a-2275d74515ba","Type":"ContainerStarted","Data":"83204f4361e67cac22dd41f9d592b7dca4f10690dc1e337a44bd72c50deacef9"}
Sep 18 19:38:37.062119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.062063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"924245df7df8fde79a0c907328f37e5a1f54170df7ed7223a1beb5df0dc4cbc2"}
Sep 18 19:38:37.063051 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.063017    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:37.078043 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.077978    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbzvr" podStartSLOduration=4.823783963 podStartE2EDuration="25.07795418s" totalImagesPullingTime="20.254170217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:37.077505691 +0000 UTC m=+24.679868873" watchObservedRunningTime="2026-09-18 19:38:37.07795418 +0000 UTC m=+24.680317365"
Sep 18 19:38:37.162598 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.162514    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:37.264894 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.264827    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:37.548627 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.548540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:37.921970 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.921876    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:37.922155 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:37.922006    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:37.922155 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:37.922067    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:37.922264 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:37.922177    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:38.147189 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:38.147154    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:38.467271 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:38.467227    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:38.921942 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:38.921906    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:38.922129 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:38.922030    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:38.951500 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:38.951464    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:39.340150 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:39.339977    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:39.559303 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:39.559273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:39.747366 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:39.747275    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:39.922305 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:39.922270    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:39.922479 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:39.922382    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:39.922479 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:39.922437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:39.922566 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:39.922545    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:39.947220 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:39.947184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:40.073119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.073025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" event={"ID":"05f4058e-a32c-473e-b179-158080388120","Type":"ContainerStarted","Data":"0e32d1fd46a6b471a7ebbbdd34ca4322342c99221c9a235f90e17fd3bdb0f369"}
Sep 18 19:38:40.073310 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.073291    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:40.073382 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.073365    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:40.073423 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.073407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:40.075214 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.075184    2560 generic.go:356] "Generic (PLEG): container finished" podID="5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6" containerID="f1dc1adce50e9dd1be32188eeda7fb8b10cbffff91acf89186b5c657cdde7c1a" exitCode=0
Sep 18 19:38:40.075330 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.075234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" event={"ID":"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6","Type":"ContainerDied","Data":"f1dc1adce50e9dd1be32188eeda7fb8b10cbffff91acf89186b5c657cdde7c1a"}
Sep 18 19:38:40.089354 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.089324    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:40.090002 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.089975    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:38:40.105130 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.105078    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m748s" podStartSLOduration=9.931605201 podStartE2EDuration="28.10506476s" totalImagesPullingTime="18.173459559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:40.10491047 +0000 UTC m=+27.707273652" watchObservedRunningTime="2026-09-18 19:38:40.10506476 +0000 UTC m=+27.707427942"
Sep 18 19:38:40.542566 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.542474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:40.922467 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.922379    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:40.922612 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:40.922519    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:40.948974 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:40.948936    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:41.005860 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:41.005826    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:41.144098 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:41.144068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:41.548945 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:41.548905    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:41.752243 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:41.752206    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:41.852874 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:41.852762    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:41.921979 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:41.921945    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:41.922140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:41.921945    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:41.922140 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:41.922063    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:41.922140 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:41.922114    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:41.953459 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:41.953428    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:42.080763 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:42.080732    2560 generic.go:356] "Generic (PLEG): container finished" podID="5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6" containerID="dc6ae8515a41a47beca7c5294b2f22274a8a73e949f1ed8837c8d21a0cb0e063" exitCode=0
Sep 18 19:38:42.080943 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:42.080808    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" event={"ID":"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6","Type":"ContainerDied","Data":"dc6ae8515a41a47beca7c5294b2f22274a8a73e949f1ed8837c8d21a0cb0e063"}
Sep 18 19:38:42.346095 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:42.346001    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:42.743314 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:42.743282    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:42.922729 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:42.922697    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:42.922928 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:42.922785    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:43.085747 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:43.085647    2560 generic.go:356] "Generic (PLEG): container finished" podID="5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6" containerID="5a34b25efc00286fd4665521e96e17984f2ceabea1da8b13c5f8d82ce78151b1" exitCode=0
Sep 18 19:38:43.085747 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:43.085699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" event={"ID":"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6","Type":"ContainerDied","Data":"5a34b25efc00286fd4665521e96e17984f2ceabea1da8b13c5f8d82ce78151b1"}
Sep 18 19:38:43.144840 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:43.144794    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:43.344372 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:43.344274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:43.744471 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:43.744378    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:43.921891 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:43.921843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:43.922071 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:43.921975    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:43.922071 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:43.922027    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:43.922189 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:43.922131    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:44.152129 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:44.152042    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:44.343296 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:44.343257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:44.922787 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:44.922748    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:44.923376 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:44.922902    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:44.948841 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:44.948790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:45.143314 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:45.143285    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:45.343687 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:45.343607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:45.742681 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:45.742602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:45.922433 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:45.922398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:45.922619 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:45.922440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:45.922619 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:45.922532    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:45.922742 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:45.922684    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:46.542712 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:46.542669    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:46.581820 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:46.581788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:46.581820 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:46.581831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:46.582070 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.581968    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:46.582070 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.582034    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret podName:d6972273-7829-45b0-9158-bfada0b133e1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:18.582014496 +0000 UTC m=+66.184377656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret") pod "global-pull-secret-syncer-d579p" (UID: "d6972273-7829-45b0-9158-bfada0b133e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:46.582070 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.581968    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:46.582235 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.582146    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs podName:d583d577-4430-4736-9cff-606fe6ee92e9 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:18.582124357 +0000 UTC m=+66.184487534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs") pod "network-metrics-daemon-6v4ch" (UID: "d583d577-4430-4736-9cff-606fe6ee92e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:46.682580 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:46.682545    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:46.682741 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.682707    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:46.682741 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.682729    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:46.682741 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.682739    2560 projected.go:196] Error preparing data for projected volume kube-api-access-nzfx6 for pod openshift-network-diagnostics/network-check-target-qvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:46.682841 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.682790    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6 podName:e6c68d39-1394-4fea-969f-422a6817eb6b nodeName:}" failed. No retries permitted until 2026-09-18 19:39:18.682775862 +0000 UTC m=+66.285139025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzfx6" (UniqueName: "kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6") pod "network-check-target-qvw9h" (UID: "e6c68d39-1394-4fea-969f-422a6817eb6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:46.745157 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:46.745118    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:46.922398 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:46.922310    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:46.922533 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:46.922429    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:46.941286 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:46.941253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:47.148950 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:47.148912    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:47.922522 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:47.922482    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:47.923033 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:47.922486    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:47.923033 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:47.922613    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:47.923033 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:47.922671    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:48.101036 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:48.100998    2560 generic.go:356] "Generic (PLEG): container finished" podID="5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6" containerID="e30842e1a1ffd5754284febdaa1a89da9211f7e54ca609bd9e1f6ddd70d3b8c3" exitCode=0
Sep 18 19:38:48.101201 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:48.101057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" event={"ID":"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6","Type":"ContainerDied","Data":"e30842e1a1ffd5754284febdaa1a89da9211f7e54ca609bd9e1f6ddd70d3b8c3"}
Sep 18 19:38:48.922319 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:48.922283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:48.922515 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:48.922392    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:49.106991 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:49.106958    2560 generic.go:356] "Generic (PLEG): container finished" podID="5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6" containerID="98e86fcb44ac9c58152e1d680392a2e892e9afbd0dbbd638cd035e787c204254" exitCode=0
Sep 18 19:38:49.107356 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:49.107011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" event={"ID":"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6","Type":"ContainerDied","Data":"98e86fcb44ac9c58152e1d680392a2e892e9afbd0dbbd638cd035e787c204254"}
Sep 18 19:38:49.922209 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:49.922014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:49.922383 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:49.922035    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:49.922383 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:49.922290    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:49.922383 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:49.922350    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:50.112387 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:50.112348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" event={"ID":"5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6","Type":"ContainerStarted","Data":"cdd00fb53d1fa2522fbd8800b2686b4d9a4a1f8f015f827279293fa683a4d55e"}
Sep 18 19:38:50.133498 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:50.133445    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zjk6b" podStartSLOduration=9.884375493 podStartE2EDuration="38.13342792s" totalImagesPullingTime="28.249052427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:50.13242579 +0000 UTC m=+37.734788971" watchObservedRunningTime="2026-09-18 19:38:50.13342792 +0000 UTC m=+37.735791102"
Sep 18 19:38:50.922287 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:50.922249    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:50.922484 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:50.922367    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:51.922049 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:51.922009    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:51.922505 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:51.922009    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:51.924896 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:51.922824    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:51.924896 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:51.923010    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:52.921427 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:52.921391    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:52.922408 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:52.922392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:52.922709 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:52.922474    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:53.042272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:53.042237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:53.144038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:53.144003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:53.236047 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:53.235962    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:53.335225 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:53.335186    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:53.922163 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:53.922135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:53.922316 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:53.922135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:53.922316 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:53.922244    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:53.922382 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:53.922311    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:54.109238 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:54.109201    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d579p"]
Sep 18 19:38:54.109722 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:54.109319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:54.109722 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:54.109422    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:54.110056 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:54.109966    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6v4ch"]
Sep 18 19:38:54.110691 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:54.110665    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvw9h"]
Sep 18 19:38:54.121720 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:54.121646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:54.121890 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:54.121770    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:54.121890 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:54.121875    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:54.122030 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:54.121974    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:55.922169 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:55.922118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:55.922629 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:55.922247    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:55.922629 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:55.922260    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:55.922629 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:55.922278    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:55.922629 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:55.922359    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:55.922629 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:55.922488    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:57.922701 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:57.922663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:57.923199 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:57.922671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:57.923199 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:57.922769    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d579p" podUID="d6972273-7829-45b0-9158-bfada0b133e1"
Sep 18 19:38:57.923199 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:57.922671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:57.923199 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:57.922871    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvw9h" podUID="e6c68d39-1394-4fea-969f-422a6817eb6b"
Sep 18 19:38:57.923199 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:38:57.922953    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6v4ch" podUID="d583d577-4430-4736-9cff-606fe6ee92e9"
Sep 18 19:38:58.538424 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.538394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:58.642733 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.642703    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:58.655257 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.655232    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-19.ec2.internal" event="NodeReady"
Sep 18 19:38:58.655407 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.655380    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:38:58.697732 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.697697    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bttxn"]
Sep 18 19:38:58.729516 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.729485    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-42fvk"]
Sep 18 19:38:58.729672 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.729613    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.731894 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.731871    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:38:58.732097 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.732082    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:38:58.732175 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.732159    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j2prk\""
Sep 18 19:38:58.743557 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.743413    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bttxn"]
Sep 18 19:38:58.743656 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.743573    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42fvk"]
Sep 18 19:38:58.743656 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.743519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42fvk"
Sep 18 19:38:58.748810 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.746452    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hn8v8\""
Sep 18 19:38:58.748810 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.747171    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:38:58.748810 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.747446    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:38:58.748810 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.748027    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:38:58.771498 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.771465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcj7l\" (UniqueName: \"kubernetes.io/projected/95b30c4f-7580-44ba-a3f9-229180da41d5-kube-api-access-pcj7l\") pod \"ingress-canary-42fvk\" (UID: \"95b30c4f-7580-44ba-a3f9-229180da41d5\") " pod="openshift-ingress-canary/ingress-canary-42fvk"
Sep 18 19:38:58.771664 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.771511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77157d0e-1282-4af5-95c3-8a63614e1dbc-metrics-tls\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.771664 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.771544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhjd\" (UniqueName: \"kubernetes.io/projected/77157d0e-1282-4af5-95c3-8a63614e1dbc-kube-api-access-vwhjd\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.771664 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.771587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77157d0e-1282-4af5-95c3-8a63614e1dbc-tmp-dir\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.771664 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.771636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95b30c4f-7580-44ba-a3f9-229180da41d5-cert\") pod \"ingress-canary-42fvk\" (UID: \"95b30c4f-7580-44ba-a3f9-229180da41d5\") " pod="openshift-ingress-canary/ingress-canary-42fvk"
Sep 18 19:38:58.771664 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.771653    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77157d0e-1282-4af5-95c3-8a63614e1dbc-config-volume\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.819528 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.819480    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:58.872484 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.872447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77157d0e-1282-4af5-95c3-8a63614e1dbc-metrics-tls\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.872484 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.872487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhjd\" (UniqueName: \"kubernetes.io/projected/77157d0e-1282-4af5-95c3-8a63614e1dbc-kube-api-access-vwhjd\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.872777 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.872753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77157d0e-1282-4af5-95c3-8a63614e1dbc-tmp-dir\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.872878 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.872834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95b30c4f-7580-44ba-a3f9-229180da41d5-cert\") pod \"ingress-canary-42fvk\" (UID: \"95b30c4f-7580-44ba-a3f9-229180da41d5\") " pod="openshift-ingress-canary/ingress-canary-42fvk"
Sep 18 19:38:58.872928 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.872893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77157d0e-1282-4af5-95c3-8a63614e1dbc-config-volume\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.872928 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.872917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcj7l\" (UniqueName: \"kubernetes.io/projected/95b30c4f-7580-44ba-a3f9-229180da41d5-kube-api-access-pcj7l\") pod \"ingress-canary-42fvk\" (UID: \"95b30c4f-7580-44ba-a3f9-229180da41d5\") " pod="openshift-ingress-canary/ingress-canary-42fvk"
Sep 18 19:38:58.873155 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.873131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77157d0e-1282-4af5-95c3-8a63614e1dbc-tmp-dir\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.873454 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.873432    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77157d0e-1282-4af5-95c3-8a63614e1dbc-config-volume\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.876505 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.876481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77157d0e-1282-4af5-95c3-8a63614e1dbc-metrics-tls\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.876612 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.876584    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95b30c4f-7580-44ba-a3f9-229180da41d5-cert\") pod \"ingress-canary-42fvk\" (UID: \"95b30c4f-7580-44ba-a3f9-229180da41d5\") " pod="openshift-ingress-canary/ingress-canary-42fvk"
Sep 18 19:38:58.880308 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.880263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhjd\" (UniqueName: \"kubernetes.io/projected/77157d0e-1282-4af5-95c3-8a63614e1dbc-kube-api-access-vwhjd\") pod \"dns-default-bttxn\" (UID: \"77157d0e-1282-4af5-95c3-8a63614e1dbc\") " pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:58.880444 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:58.880411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcj7l\" (UniqueName: \"kubernetes.io/projected/95b30c4f-7580-44ba-a3f9-229180da41d5-kube-api-access-pcj7l\") pod \"ingress-canary-42fvk\" (UID: \"95b30c4f-7580-44ba-a3f9-229180da41d5\") " pod="openshift-ingress-canary/ingress-canary-42fvk"
Sep 18 19:38:59.039487 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.039395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bttxn"
Sep 18 19:38:59.053566 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.053536    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42fvk"
Sep 18 19:38:59.385085 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.384992    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bttxn"]
Sep 18 19:38:59.385696 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.385673    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42fvk"]
Sep 18 19:38:59.388581 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:59.388554    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77157d0e_1282_4af5_95c3_8a63614e1dbc.slice/crio-71b00b514a7cacf8e83f748b9ee767cb6c45b9b83e0fd3bbace4cdfea11188a7 WatchSource:0}: Error finding container 71b00b514a7cacf8e83f748b9ee767cb6c45b9b83e0fd3bbace4cdfea11188a7: Status 404 returned error can't find the container with id 71b00b514a7cacf8e83f748b9ee767cb6c45b9b83e0fd3bbace4cdfea11188a7
Sep 18 19:38:59.389328 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:38:59.389265    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b30c4f_7580_44ba_a3f9_229180da41d5.slice/crio-0dc153a67273ece5155d63acb4627c961ba41eae0da411001ab80691ce9ed53f WatchSource:0}: Error finding container 0dc153a67273ece5155d63acb4627c961ba41eae0da411001ab80691ce9ed53f: Status 404 returned error can't find the container with id 0dc153a67273ece5155d63acb4627c961ba41eae0da411001ab80691ce9ed53f
Sep 18 19:38:59.922372 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.921901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:38:59.922372 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.921901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:38:59.922372 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.921907    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:38:59.926126 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.925926    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9245\""
Sep 18 19:38:59.926126 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.926124    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66vm6\""
Sep 18 19:38:59.926478 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.926459    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:38:59.926941 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.926671    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:38:59.926941 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.926731    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:38:59.926941 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:38:59.926904    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:00.138897 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:00.138837    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42fvk" event={"ID":"95b30c4f-7580-44ba-a3f9-229180da41d5","Type":"ContainerStarted","Data":"0dc153a67273ece5155d63acb4627c961ba41eae0da411001ab80691ce9ed53f"}
Sep 18 19:39:00.140348 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:00.140298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bttxn" event={"ID":"77157d0e-1282-4af5-95c3-8a63614e1dbc","Type":"ContainerStarted","Data":"71b00b514a7cacf8e83f748b9ee767cb6c45b9b83e0fd3bbace4cdfea11188a7"}
Sep 18 19:39:02.630202 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:02.629949    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:02.704894 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:02.704836    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:03.148446 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:03.148414    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42fvk" event={"ID":"95b30c4f-7580-44ba-a3f9-229180da41d5","Type":"ContainerStarted","Data":"521fd33dc657885fac926971512d16c72457cda6e0252889f4516b5be0210795"}
Sep 18 19:39:03.150134 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:03.150108    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bttxn" event={"ID":"77157d0e-1282-4af5-95c3-8a63614e1dbc","Type":"ContainerStarted","Data":"82b62e9e849b703606afbf94802c7c3c9b1d62ae00b2ac7632d2e7ba9915a22d"}
Sep 18 19:39:03.150258 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:03.150161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bttxn" event={"ID":"77157d0e-1282-4af5-95c3-8a63614e1dbc","Type":"ContainerStarted","Data":"51187b17afd70068c66ee04595785fc036ae2ff8e43936dbd9ab56dd10cb8c6a"}
Sep 18 19:39:03.150258 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:03.150184    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bttxn"
Sep 18 19:39:03.168952 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:03.168892    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-42fvk" podStartSLOduration=2.379470791 podStartE2EDuration="5.168876619s" totalImagesPullingTime="2.789405828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:03.16796228 +0000 UTC m=+50.770325465" watchObservedRunningTime="2026-09-18 19:39:03.168876619 +0000 UTC m=+50.771239794"
Sep 18 19:39:03.184340 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:03.184290    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bttxn" podStartSLOduration=2.402664825 podStartE2EDuration="5.184272728s" totalImagesPullingTime="2.781607903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:03.183680992 +0000 UTC m=+50.786044173" watchObservedRunningTime="2026-09-18 19:39:03.184272728 +0000 UTC m=+50.786635911"
Sep 18 19:39:06.586467 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.586424    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:06.686700 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.686667    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:06.787589 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.787554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:06.854056 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.853972    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"]
Sep 18 19:39:06.855732 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.855708    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"
Sep 18 19:39:06.856365 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.856338    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:06.858219 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.858204    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 19:39:06.858284 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.858262    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-j4sqc\""
Sep 18 19:39:06.864143 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.864120    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"]
Sep 18 19:39:06.929429 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.929395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/472ca6b6-4435-4e36-b0c7-19e1069d8d6d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2tgcf\" (UID: \"472ca6b6-4435-4e36-b0c7-19e1069d8d6d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"
Sep 18 19:39:06.960927 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:06.960893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.030213 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.030170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/472ca6b6-4435-4e36-b0c7-19e1069d8d6d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2tgcf\" (UID: \"472ca6b6-4435-4e36-b0c7-19e1069d8d6d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"
Sep 18 19:39:07.033940 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.033912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/472ca6b6-4435-4e36-b0c7-19e1069d8d6d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2tgcf\" (UID: \"472ca6b6-4435-4e36-b0c7-19e1069d8d6d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"
Sep 18 19:39:07.133383 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.133301    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.165376 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.165341    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"
Sep 18 19:39:07.190287 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.190242    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.275611 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.275574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.338494 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.338451    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.422112 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.422080    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.437737 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.437707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"]
Sep 18 19:39:07.441107 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:07.441075    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472ca6b6_4435_4e36_b0c7_19e1069d8d6d.slice/crio-19503aa7a864199d8b1fe43ecc81f3a83ed22ca5e6c3a4d3635622d0da49785f WatchSource:0}: Error finding container 19503aa7a864199d8b1fe43ecc81f3a83ed22ca5e6c3a4d3635622d0da49785f: Status 404 returned error can't find the container with id 19503aa7a864199d8b1fe43ecc81f3a83ed22ca5e6c3a4d3635622d0da49785f
Sep 18 19:39:07.462298 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.462265    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-btdg5"]
Sep 18 19:39:07.464326 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.464310    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-btdg5"
Sep 18 19:39:07.466808 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.466778    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:39:07.467119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.467100    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:39:07.467199 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.467150    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bmwzp\""
Sep 18 19:39:07.475280 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.475260    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-btdg5"]
Sep 18 19:39:07.500786 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.500756    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.534916 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.534883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txlkm\" (UniqueName: \"kubernetes.io/projected/84ffd9ed-0328-4d69-bcdd-1e858ca1bcee-kube-api-access-txlkm\") pod \"downloads-6594f4c75-btdg5\" (UID: \"84ffd9ed-0328-4d69-bcdd-1e858ca1bcee\") " pod="openshift-console/downloads-6594f4c75-btdg5"
Sep 18 19:39:07.565915 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.565880    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-48hgl"]
Sep 18 19:39:07.567720 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.567703    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.570083 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.570045    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:39:07.570182 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.570045    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:39:07.570245 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.570228    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:39:07.570357 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.570340    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:39:07.570625 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.570609    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wglrh\""
Sep 18 19:39:07.578630 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.578603    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48hgl"]
Sep 18 19:39:07.589526 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.589496    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.635613 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.635574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.635613 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.635621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d37679c8-1c35-4052-942d-624cb5ebe193-data-volume\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.635830 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.635763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txlkm\" (UniqueName: \"kubernetes.io/projected/84ffd9ed-0328-4d69-bcdd-1e858ca1bcee-kube-api-access-txlkm\") pod \"downloads-6594f4c75-btdg5\" (UID: \"84ffd9ed-0328-4d69-bcdd-1e858ca1bcee\") " pod="openshift-console/downloads-6594f4c75-btdg5"
Sep 18 19:39:07.635830 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.635807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.635830 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.635827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d37679c8-1c35-4052-942d-624cb5ebe193-crio-socket\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.635966 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.635869    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8jn\" (UniqueName: \"kubernetes.io/projected/d37679c8-1c35-4052-942d-624cb5ebe193-kube-api-access-zj8jn\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.643690 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.643665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txlkm\" (UniqueName: \"kubernetes.io/projected/84ffd9ed-0328-4d69-bcdd-1e858ca1bcee-kube-api-access-txlkm\") pod \"downloads-6594f4c75-btdg5\" (UID: \"84ffd9ed-0328-4d69-bcdd-1e858ca1bcee\") " pod="openshift-console/downloads-6594f4c75-btdg5"
Sep 18 19:39:07.737492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.737359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8jn\" (UniqueName: \"kubernetes.io/projected/d37679c8-1c35-4052-942d-624cb5ebe193-kube-api-access-zj8jn\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.737492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.737423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.737793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d37679c8-1c35-4052-942d-624cb5ebe193-data-volume\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:07.737964    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.738023    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d37679c8-1c35-4052-942d-624cb5ebe193-data-volume\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.738150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.738185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d37679c8-1c35-4052-942d-624cb5ebe193-crio-socket\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:07.738230    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls podName:d37679c8-1c35-4052-942d-624cb5ebe193 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:08.238204856 +0000 UTC m=+55.840568019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48hgl" (UID: "d37679c8-1c35-4052-942d-624cb5ebe193") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:07.738301    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:07.738361    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm podName:d37679c8-1c35-4052-942d-624cb5ebe193 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:08.238342498 +0000 UTC m=+55.840705677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48hgl" (UID: "d37679c8-1c35-4052-942d-624cb5ebe193") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:07.740320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.738371    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d37679c8-1c35-4052-942d-624cb5ebe193-crio-socket\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.746272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.746247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8jn\" (UniqueName: \"kubernetes.io/projected/d37679c8-1c35-4052-942d-624cb5ebe193-kube-api-access-zj8jn\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:07.772990 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.772958    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-btdg5"
Sep 18 19:39:07.844595 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.844559    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:07.942584 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:07.942551    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.042327 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:08.042219    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-btdg5"]
Sep 18 19:39:08.044492 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:08.044462    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ffd9ed_0328_4d69_bcdd_1e858ca1bcee.slice/crio-cedb45bb9fc761c351ff8f923e86850affd47b7cb4c599f4883f485ff99ce89a WatchSource:0}: Error finding container cedb45bb9fc761c351ff8f923e86850affd47b7cb4c599f4883f485ff99ce89a: Status 404 returned error can't find the container with id cedb45bb9fc761c351ff8f923e86850affd47b7cb4c599f4883f485ff99ce89a
Sep 18 19:39:08.136387 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:08.136354    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bttxn_77157d0e-1282-4af5-95c3-8a63614e1dbc/dns/0.log"
Sep 18 19:39:08.163642 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:08.163600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf" event={"ID":"472ca6b6-4435-4e36-b0c7-19e1069d8d6d","Type":"ContainerStarted","Data":"19503aa7a864199d8b1fe43ecc81f3a83ed22ca5e6c3a4d3635622d0da49785f"}
Sep 18 19:39:08.164945 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:08.164917    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-btdg5" event={"ID":"84ffd9ed-0328-4d69-bcdd-1e858ca1bcee","Type":"ContainerStarted","Data":"cedb45bb9fc761c351ff8f923e86850affd47b7cb4c599f4883f485ff99ce89a"}
Sep 18 19:39:08.241547 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:08.241509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:08.241718 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:08.241571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:08.241718 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:08.241632    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:08.241718 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:08.241691    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:08.241718 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:08.241709    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm podName:d37679c8-1c35-4052-942d-624cb5ebe193 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.241687469 +0000 UTC m=+56.844050631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48hgl" (UID: "d37679c8-1c35-4052-942d-624cb5ebe193") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:08.241888 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:08.241728    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls podName:d37679c8-1c35-4052-942d-624cb5ebe193 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.241717851 +0000 UTC m=+56.844081022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48hgl" (UID: "d37679c8-1c35-4052-942d-624cb5ebe193") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:08.336897 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:08.336798    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bttxn_77157d0e-1282-4af5-95c3-8a63614e1dbc/kube-rbac-proxy/0.log"
Sep 18 19:39:08.948421 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:08.948369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.158547 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.158507    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.169437 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.169403    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf" event={"ID":"472ca6b6-4435-4e36-b0c7-19e1069d8d6d","Type":"ContainerStarted","Data":"47df8b0d99d3e511f8e9339ae2d44880e01a4f196f903cbad228e3b33322499f"}
Sep 18 19:39:09.169642 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.169620    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"
Sep 18 19:39:09.175068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.175038    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf"
Sep 18 19:39:09.186252 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.186196    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2tgcf" podStartSLOduration=2.149047184 podStartE2EDuration="3.186177191s" totalImagesPullingTime="1.037130007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:09.185286825 +0000 UTC m=+56.787650009" watchObservedRunningTime="2026-09-18 19:39:09.186177191 +0000 UTC m=+56.788540376"
Sep 18 19:39:09.234416 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.234328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.248913 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.248885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:09.249092 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.249012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:09.249092 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:09.249034    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:09.249212 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:09.249105    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls podName:d37679c8-1c35-4052-942d-624cb5ebe193 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:11.249085316 +0000 UTC m=+58.851448492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48hgl" (UID: "d37679c8-1c35-4052-942d-624cb5ebe193") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:09.249212 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:09.249109    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.249212 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:09.249167    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm podName:d37679c8-1c35-4052-942d-624cb5ebe193 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:11.249150764 +0000 UTC m=+58.851513937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48hgl" (UID: "d37679c8-1c35-4052-942d-624cb5ebe193") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.359527 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.359488    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.460501 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.460459    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.560958 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.560866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.661623 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.661590    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.736223 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.736195    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6gg2_d8cbf7a2-e0da-417e-8353-9397e53a62d0/dns-node-resolver/0.log"
Sep 18 19:39:09.885451 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.885310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.912930 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.912891    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"]
Sep 18 19:39:09.936741 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.936711    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"]
Sep 18 19:39:09.936932 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.936834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:09.940181 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.940150    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:39:09.940326 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.940273    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:39:09.940326 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.940300    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 19:39:09.941516 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.941495    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jl8hc\""
Sep 18 19:39:09.941683 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.941660    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:39:09.941783 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.941751    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 19:39:09.953879 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.953832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:09.954250 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.953972    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:09.954250 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.954013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5275c10-10f9-457d-82bb-ee79691f6a5f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:09.954250 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.954039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvbp\" (UniqueName: \"kubernetes.io/projected/f5275c10-10f9-457d-82bb-ee79691f6a5f-kube-api-access-jjvbp\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:09.956593 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:09.956564    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:10.054933 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.054895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.055105 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.055019    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5275c10-10f9-457d-82bb-ee79691f6a5f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.055105 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.055047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvbp\" (UniqueName: \"kubernetes.io/projected/f5275c10-10f9-457d-82bb-ee79691f6a5f-kube-api-access-jjvbp\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.055105 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.055090    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.055225 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:10.055211    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 18 19:39:10.055278 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:10.055268    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-tls podName:f5275c10-10f9-457d-82bb-ee79691f6a5f nodeName:}" failed. No retries permitted until 2026-09-18 19:39:10.555254718 +0000 UTC m=+58.157617879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-bfjxg" (UID: "f5275c10-10f9-457d-82bb-ee79691f6a5f") : secret "prometheus-operator-tls" not found
Sep 18 19:39:10.055805 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.055782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5275c10-10f9-457d-82bb-ee79691f6a5f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.057354 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.057334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.063669 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.063644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvbp\" (UniqueName: \"kubernetes.io/projected/f5275c10-10f9-457d-82bb-ee79691f6a5f-kube-api-access-jjvbp\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.064329 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.064308    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:10.162748 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.162711    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:10.271773 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.271725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:10.536455 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.536375    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7qjmw_e4555a60-1138-42ec-8931-6b4f9091edc3/node-ca/0.log"
Sep 18 19:39:10.559011 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.558969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.561562 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.561538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f5275c10-10f9-457d-82bb-ee79691f6a5f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-bfjxg\" (UID: \"f5275c10-10f9-457d-82bb-ee79691f6a5f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:10.848087 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:10.848007    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"
Sep 18 19:39:11.142909 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.142814    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg"]
Sep 18 19:39:11.146135 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:11.146102    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5275c10_10f9_457d_82bb_ee79691f6a5f.slice/crio-108e37eb2c8cf425ef8b601caf24891336218a0f4dfda459c0c35d8e6474251c WatchSource:0}: Error finding container 108e37eb2c8cf425ef8b601caf24891336218a0f4dfda459c0c35d8e6474251c: Status 404 returned error can't find the container with id 108e37eb2c8cf425ef8b601caf24891336218a0f4dfda459c0c35d8e6474251c
Sep 18 19:39:11.176797 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.176756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg" event={"ID":"f5275c10-10f9-457d-82bb-ee79691f6a5f","Type":"ContainerStarted","Data":"108e37eb2c8cf425ef8b601caf24891336218a0f4dfda459c0c35d8e6474251c"}
Sep 18 19:39:11.264818 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.264771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:11.265018 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.264835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:11.265425 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.265396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37679c8-1c35-4052-942d-624cb5ebe193-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:11.267500 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.267473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37679c8-1c35-4052-942d-624cb5ebe193-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hgl\" (UID: \"d37679c8-1c35-4052-942d-624cb5ebe193\") " pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:11.336081 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.336049    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-42fvk_95b30c4f-7580-44ba-a3f9-229180da41d5/serve-healthcheck-canary/0.log"
Sep 18 19:39:11.477188 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.477150    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48hgl"
Sep 18 19:39:11.759972 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:11.759871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48hgl"]
Sep 18 19:39:11.766693 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:11.766663    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37679c8_1c35_4052_942d_624cb5ebe193.slice/crio-bd4be8ee7f8eb4b193f0df042a920e5b5fee4979f7a925eaf39350adfac8bcc4 WatchSource:0}: Error finding container bd4be8ee7f8eb4b193f0df042a920e5b5fee4979f7a925eaf39350adfac8bcc4: Status 404 returned error can't find the container with id bd4be8ee7f8eb4b193f0df042a920e5b5fee4979f7a925eaf39350adfac8bcc4
Sep 18 19:39:12.093997 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:12.093501    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m748s"
Sep 18 19:39:12.182373 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:12.182275    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48hgl" event={"ID":"d37679c8-1c35-4052-942d-624cb5ebe193","Type":"ContainerStarted","Data":"482ff2883c374ee11e732b5979712589663bbb5bb0523d502d472057af6d38bf"}
Sep 18 19:39:12.182373 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:12.182339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48hgl" event={"ID":"d37679c8-1c35-4052-942d-624cb5ebe193","Type":"ContainerStarted","Data":"bd4be8ee7f8eb4b193f0df042a920e5b5fee4979f7a925eaf39350adfac8bcc4"}
Sep 18 19:39:12.246492 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:12.246438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:12.308193 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:12.308033    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:12.781314 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:12.781274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:13.014626 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.014532    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:13.115980 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.115917    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:13.155121 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.155085    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bttxn"
Sep 18 19:39:13.188994 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.188678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg" event={"ID":"f5275c10-10f9-457d-82bb-ee79691f6a5f","Type":"ContainerStarted","Data":"dbaa3757fa8de4b75bf6a5881ea68d83c04bd42dba6a89e142760215918ffa54"}
Sep 18 19:39:13.188994 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.188740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg" event={"ID":"f5275c10-10f9-457d-82bb-ee79691f6a5f","Type":"ContainerStarted","Data":"8abfe9240bdc2672cbd62c456ff12391344d2901bb8c7314b720a3c586d2507f"}
Sep 18 19:39:13.191669 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.191639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48hgl" event={"ID":"d37679c8-1c35-4052-942d-624cb5ebe193","Type":"ContainerStarted","Data":"8c91c04c2de56a8df07f32588770ed1b8f50eb5b6b1bb6f4dc37d8c213f5d300"}
Sep 18 19:39:13.205307 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.205147    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bfjxg" podStartSLOduration=2.87927948 podStartE2EDuration="4.205124905s" totalImagesPullingTime="1.325845425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:13.203898448 +0000 UTC m=+60.806261631" watchObservedRunningTime="2026-09-18 19:39:13.205124905 +0000 UTC m=+60.807488089"
Sep 18 19:39:13.208567 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.208522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:13.266268 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.266152    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:13.321904 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:13.321867    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.022613 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.022565    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.134436 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.134393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.202369 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.202290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48hgl" event={"ID":"d37679c8-1c35-4052-942d-624cb5ebe193","Type":"ContainerStarted","Data":"e05ce5a5c5fd3fc234307fde7001e428f641c3d53a60388d86ade0bc057b1646"}
Sep 18 19:39:15.227029 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.226978    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-48hgl" podStartSLOduration=5.960694619 podStartE2EDuration="8.226955136s" totalImagesPullingTime="2.266260517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:15.226022294 +0000 UTC m=+62.828385487" watchObservedRunningTime="2026-09-18 19:39:15.226955136 +0000 UTC m=+62.829318297"
Sep 18 19:39:15.240415 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.240377    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.272057 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.272022    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"]
Sep 18 19:39:15.275434 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.275372    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.277919 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.277831    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 19:39:15.277919 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.277833    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7dwfg\""
Sep 18 19:39:15.283159 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.283131    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:39:15.294172 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.294142    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"]
Sep 18 19:39:15.295183 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.295160    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bdnj8"]
Sep 18 19:39:15.298844 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.298551    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.306478 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.306287    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:39:15.306478 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.306334    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:39:15.306657 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.306494    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c2j4z\""
Sep 18 19:39:15.306759 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.306739    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:39:15.367210 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.367171    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.399987 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.399948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-textfile\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.400171 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400010    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.400171 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.400171 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e06902f-e997-4b56-987a-7fe7a5eb96cf-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.400171 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zsbm\" (UniqueName: \"kubernetes.io/projected/9e06902f-e997-4b56-987a-7fe7a5eb96cf-kube-api-access-8zsbm\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.400171 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-wtmp\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.400408 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-sys\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.400408 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-tls\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.400408 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400264    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-root\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.400408 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400292    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.400408 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.400408 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400382    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhv7g\" (UniqueName: \"kubernetes.io/projected/efeb9794-9860-4b1b-8877-eef83329d23b-kube-api-access-dhv7g\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.400665 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.400440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efeb9794-9860-4b1b-8877-eef83329d23b-metrics-client-ca\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.466414 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.466371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.501582 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501543    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.501754 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhv7g\" (UniqueName: \"kubernetes.io/projected/efeb9794-9860-4b1b-8877-eef83329d23b-kube-api-access-dhv7g\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.501754 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efeb9794-9860-4b1b-8877-eef83329d23b-metrics-client-ca\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.501754 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-textfile\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.501754 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.502006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e06902f-e997-4b56-987a-7fe7a5eb96cf-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.502006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zsbm\" (UniqueName: \"kubernetes.io/projected/9e06902f-e997-4b56-987a-7fe7a5eb96cf-kube-api-access-8zsbm\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.502006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-wtmp\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-sys\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-tls\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.501983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-root\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502434 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.502013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.502434 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.502059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-sys\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502434 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:15.502127    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 19:39:15.502434 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:15.502189    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-tls podName:9e06902f-e997-4b56-987a-7fe7a5eb96cf nodeName:}" failed. No retries permitted until 2026-09-18 19:39:16.002171127 +0000 UTC m=+63.604534301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-kct9k" (UID: "9e06902f-e997-4b56-987a-7fe7a5eb96cf") : secret "openshift-state-metrics-tls" not found
Sep 18 19:39:15.502434 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:15.502262    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 19:39:15.502434 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:39:15.502295    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-tls podName:efeb9794-9860-4b1b-8877-eef83329d23b nodeName:}" failed. No retries permitted until 2026-09-18 19:39:16.002283489 +0000 UTC m=+63.604646654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-tls") pod "node-exporter-bdnj8" (UID: "efeb9794-9860-4b1b-8877-eef83329d23b") : secret "node-exporter-tls" not found
Sep 18 19:39:15.502434 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.502343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-root\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502434 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.502410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-wtmp\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502865 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.502647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-textfile\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.502865 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.502709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e06902f-e997-4b56-987a-7fe7a5eb96cf-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.502865 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.502739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.503222 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.503205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efeb9794-9860-4b1b-8877-eef83329d23b-metrics-client-ca\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.504576 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.504555    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.504677 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.504635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.511300 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.511271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhv7g\" (UniqueName: \"kubernetes.io/projected/efeb9794-9860-4b1b-8877-eef83329d23b-kube-api-access-dhv7g\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:15.511418 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.511341    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zsbm\" (UniqueName: \"kubernetes.io/projected/9e06902f-e997-4b56-987a-7fe7a5eb96cf-kube-api-access-8zsbm\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:15.616695 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.616597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.719555 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.719518    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.817441 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.817407    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:15.875284 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:15.875196    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:16.006715 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.006675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-tls\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:16.007002 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.006729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:16.009430 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.009389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e06902f-e997-4b56-987a-7fe7a5eb96cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kct9k\" (UID: \"9e06902f-e997-4b56-987a-7fe7a5eb96cf\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:16.009547 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.009391    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/efeb9794-9860-4b1b-8877-eef83329d23b-node-exporter-tls\") pod \"node-exporter-bdnj8\" (UID: \"efeb9794-9860-4b1b-8877-eef83329d23b\") " pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:16.186072 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.186036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"
Sep 18 19:39:16.203611 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.203566    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:16.209155 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.209128    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bdnj8"
Sep 18 19:39:16.220759 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:16.220726    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefeb9794_9860_4b1b_8877_eef83329d23b.slice/crio-42b8eef0d1e7fbb029ffd9a3bd54685b2fb385e8959a0ba6693708db5d624d1c WatchSource:0}: Error finding container 42b8eef0d1e7fbb029ffd9a3bd54685b2fb385e8959a0ba6693708db5d624d1c: Status 404 returned error can't find the container with id 42b8eef0d1e7fbb029ffd9a3bd54685b2fb385e8959a0ba6693708db5d624d1c
Sep 18 19:39:16.307450 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.307406    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:16.380376 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.380338    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:16.477214 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.477185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k"]
Sep 18 19:39:16.479769 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:16.479739    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e06902f_e997_4b56_987a_7fe7a5eb96cf.slice/crio-5802b344a62a27cf8be92ad324dd5eb2eef78181c3d5a875f76a43585ff5023b WatchSource:0}: Error finding container 5802b344a62a27cf8be92ad324dd5eb2eef78181c3d5a875f76a43585ff5023b: Status 404 returned error can't find the container with id 5802b344a62a27cf8be92ad324dd5eb2eef78181c3d5a875f76a43585ff5023b
Sep 18 19:39:16.860370 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:16.860222    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:17.211937 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.211899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdnj8" event={"ID":"efeb9794-9860-4b1b-8877-eef83329d23b","Type":"ContainerStarted","Data":"42b8eef0d1e7fbb029ffd9a3bd54685b2fb385e8959a0ba6693708db5d624d1c"}
Sep 18 19:39:17.214199 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.214178    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k" event={"ID":"9e06902f-e997-4b56-987a-7fe7a5eb96cf","Type":"ContainerStarted","Data":"d8b8feb604eb0b2fe618de880f0d61990cb0620a833e41673ebeb256dfdab29e"}
Sep 18 19:39:17.214326 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.214229    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k" event={"ID":"9e06902f-e997-4b56-987a-7fe7a5eb96cf","Type":"ContainerStarted","Data":"5dc55c33979675a712dee77408655ea1de34987e75fd025ae0a53bf02084b21b"}
Sep 18 19:39:17.214326 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.214253    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k" event={"ID":"9e06902f-e997-4b56-987a-7fe7a5eb96cf","Type":"ContainerStarted","Data":"5802b344a62a27cf8be92ad324dd5eb2eef78181c3d5a875f76a43585ff5023b"}
Sep 18 19:39:17.610146 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.610057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:17.709981 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.709899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:17.892031 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.891957    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85f9f87b54-l6brj"]
Sep 18 19:39:17.912588 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.912555    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f9f87b54-l6brj"]
Sep 18 19:39:17.912761 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.912688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:17.915158 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.915132    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:39:17.915283 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.915160    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:39:17.915283 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.915203    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:39:17.915283 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.915209    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:39:17.916336 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.916315    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dp2h9\""
Sep 18 19:39:17.916441 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:17.916316    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:39:18.024063 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.024017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-oauth-serving-cert\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.024063 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.024065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkgh\" (UniqueName: \"kubernetes.io/projected/da143f0d-d92c-4a86-a2e7-9ab6c1212548-kube-api-access-5nkgh\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.024292 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.024094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-config\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.024292 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.024144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-service-ca\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.024292 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.024161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-oauth-config\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.024292 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.024178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-serving-cert\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.124994 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.124952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-service-ca\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.124994 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.125004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-oauth-config\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.125256 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.125095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-serving-cert\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.125256 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.125225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-oauth-serving-cert\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.125365 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.125338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkgh\" (UniqueName: \"kubernetes.io/projected/da143f0d-d92c-4a86-a2e7-9ab6c1212548-kube-api-access-5nkgh\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.125417 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.125380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-config\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.125897 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.125870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-service-ca\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.126356 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.126334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-config\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.126413 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.126365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-oauth-serving-cert\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.127911 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.127886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-oauth-config\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.128048 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.128030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-serving-cert\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.134023 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.133979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkgh\" (UniqueName: \"kubernetes.io/projected/da143f0d-d92c-4a86-a2e7-9ab6c1212548-kube-api-access-5nkgh\") pod \"console-85f9f87b54-l6brj\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") " pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.223571 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.223537    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:18.225861 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.225813    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:18.291770 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.291728    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:18.515287 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.515191    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:18.630104 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.630071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:39:18.630301 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.630140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:39:18.632636 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.632608    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:18.632778 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.632665    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:18.643914 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.643832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6972273-7829-45b0-9158-bfada0b133e1-original-pull-secret\") pod \"global-pull-secret-syncer-d579p\" (UID: \"d6972273-7829-45b0-9158-bfada0b133e1\") " pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:39:18.643914 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.643904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d583d577-4430-4736-9cff-606fe6ee92e9-metrics-certs\") pod \"network-metrics-daemon-6v4ch\" (UID: \"d583d577-4430-4736-9cff-606fe6ee92e9\") " pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:39:18.731338 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.731302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:39:18.733934 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.733905    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:18.744022 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.743998    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:18.754626 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.754603    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzfx6\" (UniqueName: \"kubernetes.io/projected/e6c68d39-1394-4fea-969f-422a6817eb6b-kube-api-access-nzfx6\") pod \"network-check-target-qvw9h\" (UID: \"e6c68d39-1394-4fea-969f-422a6817eb6b\") " pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:39:18.835872 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.835775    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d579p"
Sep 18 19:39:18.846277 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.846244    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9245\""
Sep 18 19:39:18.850229 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.850197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:18.854033 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.854006    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6v4ch"
Sep 18 19:39:18.854164 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.854107    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66vm6\""
Sep 18 19:39:18.862185 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:18.862157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:39:19.694025 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.693992    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86559f9ff-qw6qq"]
Sep 18 19:39:19.696207 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.696184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.700097 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.699943    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 19:39:19.700097 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.699961    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:39:19.700097 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.699977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 19:39:19.700097 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.699999    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1agmo7c0r8tss\""
Sep 18 19:39:19.700097 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.699951    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dfxqh\""
Sep 18 19:39:19.700097 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.700062    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 19:39:19.708287 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.708264    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86559f9ff-qw6qq"]
Sep 18 19:39:19.740521 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.740476    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-secret-metrics-server-client-certs\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.740714 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.740525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce2c667-177e-4a95-b724-a53790a29278-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.740714 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.740557    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxsx\" (UniqueName: \"kubernetes.io/projected/dce2c667-177e-4a95-b724-a53790a29278-kube-api-access-2cxsx\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.740714 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.740666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dce2c667-177e-4a95-b724-a53790a29278-audit-log\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.740714 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.740703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dce2c667-177e-4a95-b724-a53790a29278-metrics-server-audit-profiles\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.740937 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.740742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-secret-metrics-server-tls\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.740937 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.740838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-client-ca-bundle\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.841390 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.841349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce2c667-177e-4a95-b724-a53790a29278-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.841390 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.841396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxsx\" (UniqueName: \"kubernetes.io/projected/dce2c667-177e-4a95-b724-a53790a29278-kube-api-access-2cxsx\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.841634 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.841444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dce2c667-177e-4a95-b724-a53790a29278-audit-log\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.841634 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.841475    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dce2c667-177e-4a95-b724-a53790a29278-metrics-server-audit-profiles\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.841634 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.841508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-secret-metrics-server-tls\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.842075 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.841885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-client-ca-bundle\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.842075 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.841938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-secret-metrics-server-client-certs\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.842075 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.841959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dce2c667-177e-4a95-b724-a53790a29278-audit-log\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.842254 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.842208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce2c667-177e-4a95-b724-a53790a29278-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.842546 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.842523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dce2c667-177e-4a95-b724-a53790a29278-metrics-server-audit-profiles\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.844537 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.844514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-secret-metrics-server-client-certs\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.844632 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.844593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-client-ca-bundle\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.845088 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.845070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dce2c667-177e-4a95-b724-a53790a29278-secret-metrics-server-tls\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:19.849279 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:19.849259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxsx\" (UniqueName: \"kubernetes.io/projected/dce2c667-177e-4a95-b724-a53790a29278-kube-api-access-2cxsx\") pod \"metrics-server-86559f9ff-qw6qq\" (UID: \"dce2c667-177e-4a95-b724-a53790a29278\") " pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:20.008012 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.007638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:20.039325 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.039275    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:20.044463 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.044431    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"]
Sep 18 19:39:20.046629 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.046606    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:20.049031 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.049009    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 19:39:20.049176 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.049044    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n54hb\""
Sep 18 19:39:20.057568 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.057537    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"]
Sep 18 19:39:20.097023 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.096989    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:20.143993 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.143948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldlp7\" (UniqueName: \"kubernetes.io/projected/3e8c7eeb-ce95-4728-a639-81882f2b95b9-kube-api-access-ldlp7\") pod \"monitoring-plugin-5984459ff8-gjd5j\" (UID: \"3e8c7eeb-ce95-4728-a639-81882f2b95b9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:20.144193 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.144100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e8c7eeb-ce95-4728-a639-81882f2b95b9-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-gjd5j\" (UID: \"3e8c7eeb-ce95-4728-a639-81882f2b95b9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:20.158806 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.158771    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:20.245111 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.245077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldlp7\" (UniqueName: \"kubernetes.io/projected/3e8c7eeb-ce95-4728-a639-81882f2b95b9-kube-api-access-ldlp7\") pod \"monitoring-plugin-5984459ff8-gjd5j\" (UID: \"3e8c7eeb-ce95-4728-a639-81882f2b95b9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:20.245302 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.245130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e8c7eeb-ce95-4728-a639-81882f2b95b9-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-gjd5j\" (UID: \"3e8c7eeb-ce95-4728-a639-81882f2b95b9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:20.247987 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.247870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e8c7eeb-ce95-4728-a639-81882f2b95b9-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-gjd5j\" (UID: \"3e8c7eeb-ce95-4728-a639-81882f2b95b9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:20.254229 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.254202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldlp7\" (UniqueName: \"kubernetes.io/projected/3e8c7eeb-ce95-4728-a639-81882f2b95b9-kube-api-access-ldlp7\") pod \"monitoring-plugin-5984459ff8-gjd5j\" (UID: \"3e8c7eeb-ce95-4728-a639-81882f2b95b9\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:20.361226 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.361143    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:20.519749 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.519709    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d996c5bc8-czwdk"]
Sep 18 19:39:20.522108 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.522081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.524711 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.524682    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 19:39:20.524871 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.524739    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 19:39:20.524871 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.524749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 19:39:20.524871 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.524800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 19:39:20.525056 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.524930    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7vkmw\""
Sep 18 19:39:20.525193 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.525170    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 19:39:20.529514 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.529490    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 19:39:20.543946 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.543913    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d996c5bc8-czwdk"]
Sep 18 19:39:20.648009 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.647917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.648009 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.647975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-telemeter-client-tls\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.648238 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.648014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.648238 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.648043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrq86\" (UniqueName: \"kubernetes.io/projected/1639cf01-6883-409f-876e-d7510722ee5d-kube-api-access-mrq86\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.648238 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.648092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-federate-client-tls\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.648238 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.648155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-serving-certs-ca-bundle\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.648238 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.648202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-metrics-client-ca\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.648238 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.648230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-secret-telemeter-client\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.749444 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.749397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.749444 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.749448    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrq86\" (UniqueName: \"kubernetes.io/projected/1639cf01-6883-409f-876e-d7510722ee5d-kube-api-access-mrq86\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750442 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.749638    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-federate-client-tls\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750442 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.749701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-serving-certs-ca-bundle\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750442 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.749729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-metrics-client-ca\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750442 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.749763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-secret-telemeter-client\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750442 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.749823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750442 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.749879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-telemeter-client-tls\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750771 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.750600    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-serving-certs-ca-bundle\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750771 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.750618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-metrics-client-ca\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.750897 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.750784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1639cf01-6883-409f-876e-d7510722ee5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.752879 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.752583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.752879 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.752657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-federate-client-tls\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.752879 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.752780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-telemeter-client-tls\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.752879 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.752803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1639cf01-6883-409f-876e-d7510722ee5d-secret-telemeter-client\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.757388 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.757368    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrq86\" (UniqueName: \"kubernetes.io/projected/1639cf01-6883-409f-876e-d7510722ee5d-kube-api-access-mrq86\") pod \"telemeter-client-d996c5bc8-czwdk\" (UID: \"1639cf01-6883-409f-876e-d7510722ee5d\") " pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:20.768910 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.768870    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:20.818833 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.818791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:20.833546 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:20.833516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk"
Sep 18 19:39:21.366645 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:21.366601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:21.520375 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:21.520332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:25.129610 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.128931    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:25.227167 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.226948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:25.244514 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.244464    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdnj8" event={"ID":"efeb9794-9860-4b1b-8877-eef83329d23b","Type":"ContainerStarted","Data":"b4fc8f336eaf1bb133a9e7876dfd771f88d38395d60b592dc9d7e1cc068f82f7"}
Sep 18 19:39:25.248138 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.247910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k" event={"ID":"9e06902f-e997-4b56-987a-7fe7a5eb96cf","Type":"ContainerStarted","Data":"063252536e2965b9f23e5b9b12aa4acdb8d59d6bd2b96763ba598568f3451c94"}
Sep 18 19:39:25.282909 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.282827    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kct9k" podStartSLOduration=1.848817996 podStartE2EDuration="10.282805727s" totalImagesPullingTime="8.433987731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:25.282306845 +0000 UTC m=+72.884670024" watchObservedRunningTime="2026-09-18 19:39:25.282805727 +0000 UTC m=+72.885168910"
Sep 18 19:39:25.361653 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.361621    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d996c5bc8-czwdk"]
Sep 18 19:39:25.364180 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:25.364153    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1639cf01_6883_409f_876e_d7510722ee5d.slice/crio-c19b6e82c7287e7031ba3b7c7fe2de214904bc938b94906235b47a69735bef72 WatchSource:0}: Error finding container c19b6e82c7287e7031ba3b7c7fe2de214904bc938b94906235b47a69735bef72: Status 404 returned error can't find the container with id c19b6e82c7287e7031ba3b7c7fe2de214904bc938b94906235b47a69735bef72
Sep 18 19:39:25.370496 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.369434    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f9f87b54-l6brj"]
Sep 18 19:39:25.374676 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:25.374644    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda143f0d_d92c_4a86_a2e7_9ab6c1212548.slice/crio-2afa8ed9b76e69bb60f8586bc0c969236c294fd58689c37183b43f15c50899d3 WatchSource:0}: Error finding container 2afa8ed9b76e69bb60f8586bc0c969236c294fd58689c37183b43f15c50899d3: Status 404 returned error can't find the container with id 2afa8ed9b76e69bb60f8586bc0c969236c294fd58689c37183b43f15c50899d3
Sep 18 19:39:25.387304 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.387268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6v4ch"]
Sep 18 19:39:25.390125 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.390021    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"]
Sep 18 19:39:25.390296 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:25.390248    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd583d577_4430_4736_9cff_606fe6ee92e9.slice/crio-6c327d1d8e9f20cfacc16aa6b91438c711b722732c95f7130b64eb48ad608242 WatchSource:0}: Error finding container 6c327d1d8e9f20cfacc16aa6b91438c711b722732c95f7130b64eb48ad608242: Status 404 returned error can't find the container with id 6c327d1d8e9f20cfacc16aa6b91438c711b722732c95f7130b64eb48ad608242
Sep 18 19:39:25.393292 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:25.393185    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8c7eeb_ce95_4728_a639_81882f2b95b9.slice/crio-f823cf3d8e6b8ddd1f82049e146b8cbb143da6972ba3b7945ae0639f44882c50 WatchSource:0}: Error finding container f823cf3d8e6b8ddd1f82049e146b8cbb143da6972ba3b7945ae0639f44882c50: Status 404 returned error can't find the container with id f823cf3d8e6b8ddd1f82049e146b8cbb143da6972ba3b7945ae0639f44882c50
Sep 18 19:39:25.420170 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.420100    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvw9h"]
Sep 18 19:39:25.423082 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:25.423047    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c68d39_1394_4fea_969f_422a6817eb6b.slice/crio-6364c745080a546cc5ae87679795a3e87b788feb8c35d3b374ac5b44b6480e9a WatchSource:0}: Error finding container 6364c745080a546cc5ae87679795a3e87b788feb8c35d3b374ac5b44b6480e9a: Status 404 returned error can't find the container with id 6364c745080a546cc5ae87679795a3e87b788feb8c35d3b374ac5b44b6480e9a
Sep 18 19:39:25.433572 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.433545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d579p"]
Sep 18 19:39:25.436478 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:25.436450    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6972273_7829_45b0_9158_bfada0b133e1.slice/crio-09ae7e47b7bb15e51611e2f3d8f834195aadfbcb0757a8f1500ca5e8604fbe35 WatchSource:0}: Error finding container 09ae7e47b7bb15e51611e2f3d8f834195aadfbcb0757a8f1500ca5e8604fbe35: Status 404 returned error can't find the container with id 09ae7e47b7bb15e51611e2f3d8f834195aadfbcb0757a8f1500ca5e8604fbe35
Sep 18 19:39:25.437481 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.437460    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86559f9ff-qw6qq"]
Sep 18 19:39:25.440732 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:25.440709    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce2c667_177e_4a95_b724_a53790a29278.slice/crio-7e48075f55ddbf7e05c8a95369100dbd445cb0a34b7a93da02679fba48150696 WatchSource:0}: Error finding container 7e48075f55ddbf7e05c8a95369100dbd445cb0a34b7a93da02679fba48150696: Status 404 returned error can't find the container with id 7e48075f55ddbf7e05c8a95369100dbd445cb0a34b7a93da02679fba48150696
Sep 18 19:39:25.470226 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.470184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:25.532226 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:25.532159    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:26.274376 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.274335    2560 generic.go:356] "Generic (PLEG): container finished" podID="efeb9794-9860-4b1b-8877-eef83329d23b" containerID="b4fc8f336eaf1bb133a9e7876dfd771f88d38395d60b592dc9d7e1cc068f82f7" exitCode=0
Sep 18 19:39:26.274821 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.274431    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdnj8" event={"ID":"efeb9794-9860-4b1b-8877-eef83329d23b","Type":"ContainerDied","Data":"b4fc8f336eaf1bb133a9e7876dfd771f88d38395d60b592dc9d7e1cc068f82f7"}
Sep 18 19:39:26.312184 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.312030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6v4ch" event={"ID":"d583d577-4430-4736-9cff-606fe6ee92e9","Type":"ContainerStarted","Data":"6c327d1d8e9f20cfacc16aa6b91438c711b722732c95f7130b64eb48ad608242"}
Sep 18 19:39:26.317300 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.317217    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvw9h" event={"ID":"e6c68d39-1394-4fea-969f-422a6817eb6b","Type":"ContainerStarted","Data":"6364c745080a546cc5ae87679795a3e87b788feb8c35d3b374ac5b44b6480e9a"}
Sep 18 19:39:26.356883 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.341188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d579p" event={"ID":"d6972273-7829-45b0-9158-bfada0b133e1","Type":"ContainerStarted","Data":"09ae7e47b7bb15e51611e2f3d8f834195aadfbcb0757a8f1500ca5e8604fbe35"}
Sep 18 19:39:26.358099 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.357933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j" event={"ID":"3e8c7eeb-ce95-4728-a639-81882f2b95b9","Type":"ContainerStarted","Data":"f823cf3d8e6b8ddd1f82049e146b8cbb143da6972ba3b7945ae0639f44882c50"}
Sep 18 19:39:26.361590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.361505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f9f87b54-l6brj" event={"ID":"da143f0d-d92c-4a86-a2e7-9ab6c1212548","Type":"ContainerStarted","Data":"2afa8ed9b76e69bb60f8586bc0c969236c294fd58689c37183b43f15c50899d3"}
Sep 18 19:39:26.363808 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.363728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk" event={"ID":"1639cf01-6883-409f-876e-d7510722ee5d","Type":"ContainerStarted","Data":"c19b6e82c7287e7031ba3b7c7fe2de214904bc938b94906235b47a69735bef72"}
Sep 18 19:39:26.371480 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.368183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-btdg5" event={"ID":"84ffd9ed-0328-4d69-bcdd-1e858ca1bcee","Type":"ContainerStarted","Data":"aad5dfe009a5730926d9aa58504e917fb873d6bef9094ec497ff5c70d49b57ac"}
Sep 18 19:39:26.371480 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.370229    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-btdg5"
Sep 18 19:39:26.378370 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.378318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-btdg5"
Sep 18 19:39:26.380685 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.380612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq" event={"ID":"dce2c667-177e-4a95-b724-a53790a29278","Type":"ContainerStarted","Data":"7e48075f55ddbf7e05c8a95369100dbd445cb0a34b7a93da02679fba48150696"}
Sep 18 19:39:26.391486 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:26.391423    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-btdg5" podStartSLOduration=2.338102758 podStartE2EDuration="19.391402544s" totalImagesPullingTime="17.053299786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:26.389999482 +0000 UTC m=+73.992362665" watchObservedRunningTime="2026-09-18 19:39:26.391402544 +0000 UTC m=+73.993765729"
Sep 18 19:39:27.400358 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.400209    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdnj8" event={"ID":"efeb9794-9860-4b1b-8877-eef83329d23b","Type":"ContainerStarted","Data":"f0c4f6b0b288aca4cbe132eb79dbc8fd4a6d1eaf493c13aef125452e5b1c8783"}
Sep 18 19:39:27.400358 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.400271    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdnj8" event={"ID":"efeb9794-9860-4b1b-8877-eef83329d23b","Type":"ContainerStarted","Data":"1ffc061fa99ec6d557f27a8a1e19378a3a0063de68fc726a9444c432b1085f34"}
Sep 18 19:39:27.422100 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.421812    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bdnj8" podStartSLOduration=3.534522339 podStartE2EDuration="12.421788153s" totalImagesPullingTime="8.815446619s" totalInitContainerRuntime="71.819195ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:27.420371146 +0000 UTC m=+75.022734327" watchObservedRunningTime="2026-09-18 19:39:27.421788153 +0000 UTC m=+75.024151337"
Sep 18 19:39:27.564364 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.564320    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cd7496b9d-64hzj"]
Sep 18 19:39:27.587053 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.586198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd7496b9d-64hzj"]
Sep 18 19:39:27.587053 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.586325    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.608314 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.608287    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:39:27.724180 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.724092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-serving-cert\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.724180 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.724144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-trusted-ca-bundle\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.724402 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.724182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgrtc\" (UniqueName: \"kubernetes.io/projected/5b48c963-a1a2-4547-865c-550018e713d6-kube-api-access-hgrtc\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.724402 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.724207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-oauth-config\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.724402 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.724271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-service-ca\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.724402 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.724322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-oauth-serving-cert\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.724402 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.724346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-console-config\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.826550 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.825259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-serving-cert\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.826550 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.825311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-trusted-ca-bundle\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.826550 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.825348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgrtc\" (UniqueName: \"kubernetes.io/projected/5b48c963-a1a2-4547-865c-550018e713d6-kube-api-access-hgrtc\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.826550 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.825375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-oauth-config\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.826550 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.825436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-service-ca\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.826550 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.825480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-oauth-serving-cert\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.826550 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.825505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-console-config\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.829964 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.828430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-service-ca\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.829964 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.829515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-oauth-serving-cert\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.830632 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.830572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-trusted-ca-bundle\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.848253 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.840892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-console-config\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.848253 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.846553    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-serving-cert\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.861717 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.861568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-oauth-config\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.871995 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.871929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgrtc\" (UniqueName: \"kubernetes.io/projected/5b48c963-a1a2-4547-865c-550018e713d6-kube-api-access-hgrtc\") pod \"console-6cd7496b9d-64hzj\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") " pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:27.907133 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:27.906799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:29.801066 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:29.800870    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:29.862839 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:29.862785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:30.326062 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:30.325964    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:30.430532 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:30.430444    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:31.975500 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:31.975462    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:32.610838 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:32.610712    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:32.672227 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:32.672064    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:36.037805 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:36.037724    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd7496b9d-64hzj"]
Sep 18 19:39:37.015063 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:37.015026    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:37.062356 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:37.062312    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b48c963_a1a2_4547_865c_550018e713d6.slice/crio-dd994f23e6e57450e80e944c17e96a2e7387007eb348930837707d33b39a46e5 WatchSource:0}: Error finding container dd994f23e6e57450e80e944c17e96a2e7387007eb348930837707d33b39a46e5: Status 404 returned error can't find the container with id dd994f23e6e57450e80e944c17e96a2e7387007eb348930837707d33b39a46e5
Sep 18 19:39:37.070262 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:37.070234    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:37.457394 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:37.457344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd7496b9d-64hzj" event={"ID":"5b48c963-a1a2-4547-865c-550018e713d6","Type":"ContainerStarted","Data":"dd994f23e6e57450e80e944c17e96a2e7387007eb348930837707d33b39a46e5"}
Sep 18 19:39:38.466472 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.466439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6v4ch" event={"ID":"d583d577-4430-4736-9cff-606fe6ee92e9","Type":"ContainerStarted","Data":"a228f2b51af32a226351e18033b75fb82935d1fad96288113b4a255e871ceb0c"}
Sep 18 19:39:38.468112 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.468091    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvw9h" event={"ID":"e6c68d39-1394-4fea-969f-422a6817eb6b","Type":"ContainerStarted","Data":"e911a056a2ae4baf1a55e2c03a8d742bf64aa5b67ed58f356d87e5f7f7c8886d"}
Sep 18 19:39:38.470509 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.470485    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d579p" event={"ID":"d6972273-7829-45b0-9158-bfada0b133e1","Type":"ContainerStarted","Data":"362c684bfc2489ba9fab1fa4c04534825954e00cbac955e82bbc69166daecb4f"}
Sep 18 19:39:38.472928 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.472906    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j" event={"ID":"3e8c7eeb-ce95-4728-a639-81882f2b95b9","Type":"ContainerStarted","Data":"41440d9efcd834e1c4652b1fb2b5263cd04997973ac221076312c9715934a544"}
Sep 18 19:39:38.473199 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.473147    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:38.475746 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.475688    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk" event={"ID":"1639cf01-6883-409f-876e-d7510722ee5d","Type":"ContainerStarted","Data":"fb341dc8df290c249d915d77801212d6747471e11c88d76782c19fdd314ca444"}
Sep 18 19:39:38.479625 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.479601    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j"
Sep 18 19:39:38.494590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.494488    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d579p" podStartSLOduration=73.701928911 podStartE2EDuration="1m26.49446675s" totalImagesPullingTime="12.792537839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:38.49336954 +0000 UTC m=+86.095732730" watchObservedRunningTime="2026-09-18 19:39:38.49446675 +0000 UTC m=+86.096829932"
Sep 18 19:39:38.508379 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.508325    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-gjd5j" podStartSLOduration=7.96218202 podStartE2EDuration="18.508309113s" totalImagesPullingTime="10.546127093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:38.507141042 +0000 UTC m=+86.109504313" watchObservedRunningTime="2026-09-18 19:39:38.508309113 +0000 UTC m=+86.110672290"
Sep 18 19:39:38.867918 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.867815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:38.925458 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:38.925411    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:39.483248 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.483208    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6v4ch" event={"ID":"d583d577-4430-4736-9cff-606fe6ee92e9","Type":"ContainerStarted","Data":"195c91eb99523f42535d4a8ef3235b071086e27aa56d4359b619ef0bd38c5a7a"}
Sep 18 19:39:39.485536 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.485502    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f9f87b54-l6brj" event={"ID":"da143f0d-d92c-4a86-a2e7-9ab6c1212548","Type":"ContainerStarted","Data":"0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f"}
Sep 18 19:39:39.487701 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.487677    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd7496b9d-64hzj" event={"ID":"5b48c963-a1a2-4547-865c-550018e713d6","Type":"ContainerStarted","Data":"b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a"}
Sep 18 19:39:39.489573 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.489547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq" event={"ID":"dce2c667-177e-4a95-b724-a53790a29278","Type":"ContainerStarted","Data":"d4589f333c20b49824189472206476c2d9e27287e6b15f0c6b2d09e670db66b8"}
Sep 18 19:39:39.499407 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.499352    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6v4ch" podStartSLOduration=77.122780918 podStartE2EDuration="1m27.499331382s" totalImagesPullingTime="10.376550464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.498799576 +0000 UTC m=+87.101162757" watchObservedRunningTime="2026-09-18 19:39:39.499331382 +0000 UTC m=+87.101694567"
Sep 18 19:39:39.520082 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.520027    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85f9f87b54-l6brj" podStartSLOduration=9.20923338 podStartE2EDuration="22.520009283s" totalImagesPullingTime="13.310775903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.51926001 +0000 UTC m=+87.121623194" watchObservedRunningTime="2026-09-18 19:39:39.520009283 +0000 UTC m=+87.122372463"
Sep 18 19:39:39.541563 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.541502    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq" podStartSLOduration=7.753212254 podStartE2EDuration="20.541480151s" totalImagesPullingTime="12.788267897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.540373904 +0000 UTC m=+87.142737086" watchObservedRunningTime="2026-09-18 19:39:39.541480151 +0000 UTC m=+87.143843334"
Sep 18 19:39:39.559994 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.559928    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cd7496b9d-64hzj" podStartSLOduration=10.964370726 podStartE2EDuration="12.559902797s" totalImagesPullingTime="1.595532071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.558390708 +0000 UTC m=+87.160753896" watchObservedRunningTime="2026-09-18 19:39:39.559902797 +0000 UTC m=+87.162265979"
Sep 18 19:39:39.574273 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.574184    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qvw9h" podStartSLOduration=74.769673185 podStartE2EDuration="1m27.574166132s" totalImagesPullingTime="12.804492947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:39.573186581 +0000 UTC m=+87.175549764" watchObservedRunningTime="2026-09-18 19:39:39.574166132 +0000 UTC m=+87.176529314"
Sep 18 19:39:39.593800 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.593754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:39.691681 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.691646    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:39.748685 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:39.748593    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:40.008953 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.008833    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:40.008953 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.008927    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:39:40.495205 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.495157    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk" event={"ID":"1639cf01-6883-409f-876e-d7510722ee5d","Type":"ContainerStarted","Data":"0639bff3019d0b7cf7005d9fa6a352db314a908d78b4b6c510b7f6d37cb48e73"}
Sep 18 19:39:40.495667 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.495265    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk" event={"ID":"1639cf01-6883-409f-876e-d7510722ee5d","Type":"ContainerStarted","Data":"ebdfc2dc1c00d09e9969cdcb85866e86fdc3f3f29539445c02485c223ae7f899"}
Sep 18 19:39:40.516558 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.516492    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d996c5bc8-czwdk" podStartSLOduration=6.187004246 podStartE2EDuration="20.51647028s" totalImagesPullingTime="14.329466034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:40.514985727 +0000 UTC m=+88.117348909" watchObservedRunningTime="2026-09-18 19:39:40.51647028 +0000 UTC m=+88.118833462"
Sep 18 19:39:40.609147 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.609046    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85f9f87b54-l6brj"]
Sep 18 19:39:40.646651 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.646620    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d7874d8d-ndqpc"]
Sep 18 19:39:40.667016 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.666982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.667417 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.667397    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d7874d8d-ndqpc"]
Sep 18 19:39:40.754778 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.754733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-oauth-serving-cert\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.754778 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.754784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-oauth-config\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.755041 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.754821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-trusted-ca-bundle\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.755041 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.754902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-service-ca\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.755041 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.754938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-console-config\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.755041 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.754998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qlg\" (UniqueName: \"kubernetes.io/projected/0f07a55a-618a-4244-a730-e123b75df46b-kube-api-access-82qlg\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.755203 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.755054    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-serving-cert\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.855808 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.855765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-console-config\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.855808 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.855811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qlg\" (UniqueName: \"kubernetes.io/projected/0f07a55a-618a-4244-a730-e123b75df46b-kube-api-access-82qlg\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.856078 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.855833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-serving-cert\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.856078 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.855878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-oauth-serving-cert\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.856078 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.856038    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-oauth-config\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.856227 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.856121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-trusted-ca-bundle\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.856227 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.856153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-service-ca\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.856718 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.856688    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-console-config\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.856871 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.856780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-oauth-serving-cert\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.856953 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.856841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-service-ca\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.857012 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.856958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-trusted-ca-bundle\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.858443 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.858417    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-serving-cert\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.858523 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.858484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-oauth-config\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.863754 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.863711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qlg\" (UniqueName: \"kubernetes.io/projected/0f07a55a-618a-4244-a730-e123b75df46b-kube-api-access-82qlg\") pod \"console-59d7874d8d-ndqpc\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") " pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:40.976415 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:40.976376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:41.270425 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:41.270187    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d7874d8d-ndqpc"]
Sep 18 19:39:41.272969 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:39:41.272937    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f07a55a_618a_4244_a730_e123b75df46b.slice/crio-ae22fb8049eb8fb711a0c88ee9f76d41ead200c033f84d2a90b580319f80bccb WatchSource:0}: Error finding container ae22fb8049eb8fb711a0c88ee9f76d41ead200c033f84d2a90b580319f80bccb: Status 404 returned error can't find the container with id ae22fb8049eb8fb711a0c88ee9f76d41ead200c033f84d2a90b580319f80bccb
Sep 18 19:39:41.501256 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:41.501201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d7874d8d-ndqpc" event={"ID":"0f07a55a-618a-4244-a730-e123b75df46b","Type":"ContainerStarted","Data":"0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8"}
Sep 18 19:39:41.501715 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:41.501275    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d7874d8d-ndqpc" event={"ID":"0f07a55a-618a-4244-a730-e123b75df46b","Type":"ContainerStarted","Data":"ae22fb8049eb8fb711a0c88ee9f76d41ead200c033f84d2a90b580319f80bccb"}
Sep 18 19:39:41.537917 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:41.537783    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59d7874d8d-ndqpc" podStartSLOduration=1.5377583540000002 podStartE2EDuration="1.537758354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:41.536365261 +0000 UTC m=+89.138728446" watchObservedRunningTime="2026-09-18 19:39:41.537758354 +0000 UTC m=+89.140121537"
Sep 18 19:39:47.623572 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:47.623535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:47.683282 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:47.683224    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:47.908056 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:47.908019    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:47.908340 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:47.908077    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:47.912810 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:47.912782    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:48.224489 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:48.224389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:39:48.532266 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:48.532184    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:39:48.657173 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:48.657131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:48.712520 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:48.712487    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:48.858189 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:48.858106    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:48.915980 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:48.915939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:49.490510 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:49.490469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:39:50.977195 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:50.977155    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:50.977570 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:50.977223    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:50.981936 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:50.981911    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:51.543064 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:51.543031    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:39:51.585685 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:51.585649    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd7496b9d-64hzj"]
Sep 18 19:39:51.681290 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:51.681251    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:51.782015 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:51.781983    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:51.849105 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:39:51.849031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:00.014064 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:00.014033    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:40:00.017798 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:00.017774    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86559f9ff-qw6qq"
Sep 18 19:40:02.619170 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:02.619135    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:02.682935 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:02.682905    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:06.522232 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.522192    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85f9f87b54-l6brj" podUID="da143f0d-d92c-4a86-a2e7-9ab6c1212548" containerName="console" containerID="cri-o://0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f" gracePeriod=15
Sep 18 19:40:06.799131 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.799047    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:40:06.868349 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.868312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-oauth-config\") pod \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") "
Sep 18 19:40:06.868556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.868373    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-service-ca\") pod \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") "
Sep 18 19:40:06.868556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.868432    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-oauth-serving-cert\") pod \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") "
Sep 18 19:40:06.868556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.868458    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-config\") pod \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") "
Sep 18 19:40:06.868556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.868492    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-serving-cert\") pod \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") "
Sep 18 19:40:06.868556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.868521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nkgh\" (UniqueName: \"kubernetes.io/projected/da143f0d-d92c-4a86-a2e7-9ab6c1212548-kube-api-access-5nkgh\") pod \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\" (UID: \"da143f0d-d92c-4a86-a2e7-9ab6c1212548\") "
Sep 18 19:40:06.868934 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.868902    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da143f0d-d92c-4a86-a2e7-9ab6c1212548" (UID: "da143f0d-d92c-4a86-a2e7-9ab6c1212548"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:06.869033 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.868906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-service-ca" (OuterVolumeSpecName: "service-ca") pod "da143f0d-d92c-4a86-a2e7-9ab6c1212548" (UID: "da143f0d-d92c-4a86-a2e7-9ab6c1212548"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:06.869033 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.869012    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-config" (OuterVolumeSpecName: "console-config") pod "da143f0d-d92c-4a86-a2e7-9ab6c1212548" (UID: "da143f0d-d92c-4a86-a2e7-9ab6c1212548"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:06.870835 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.870805    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da143f0d-d92c-4a86-a2e7-9ab6c1212548" (UID: "da143f0d-d92c-4a86-a2e7-9ab6c1212548"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:06.870957 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.870869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da143f0d-d92c-4a86-a2e7-9ab6c1212548" (UID: "da143f0d-d92c-4a86-a2e7-9ab6c1212548"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:06.870957 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.870869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da143f0d-d92c-4a86-a2e7-9ab6c1212548-kube-api-access-5nkgh" (OuterVolumeSpecName: "kube-api-access-5nkgh") pod "da143f0d-d92c-4a86-a2e7-9ab6c1212548" (UID: "da143f0d-d92c-4a86-a2e7-9ab6c1212548"). InnerVolumeSpecName "kube-api-access-5nkgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:06.969562 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.969518    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-service-ca\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:06.969562 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.969551    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-oauth-serving-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:06.969562 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.969561    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:06.969562 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.969570    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-serving-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:06.969831 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.969578    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkgh\" (UniqueName: \"kubernetes.io/projected/da143f0d-d92c-4a86-a2e7-9ab6c1212548-kube-api-access-5nkgh\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:06.969831 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:06.969586    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da143f0d-d92c-4a86-a2e7-9ab6c1212548-console-oauth-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:07.598534 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.598496    2560 generic.go:356] "Generic (PLEG): container finished" podID="da143f0d-d92c-4a86-a2e7-9ab6c1212548" containerID="0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f" exitCode=0
Sep 18 19:40:07.599001 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.598560    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f9f87b54-l6brj"
Sep 18 19:40:07.599001 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.598584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f9f87b54-l6brj" event={"ID":"da143f0d-d92c-4a86-a2e7-9ab6c1212548","Type":"ContainerDied","Data":"0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f"}
Sep 18 19:40:07.599001 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.598662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f9f87b54-l6brj" event={"ID":"da143f0d-d92c-4a86-a2e7-9ab6c1212548","Type":"ContainerDied","Data":"2afa8ed9b76e69bb60f8586bc0c969236c294fd58689c37183b43f15c50899d3"}
Sep 18 19:40:07.599001 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.598682    2560 scope.go:117] "RemoveContainer" containerID="0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f"
Sep 18 19:40:07.606739 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.606717    2560 scope.go:117] "RemoveContainer" containerID="0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f"
Sep 18 19:40:07.607041 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:40:07.607012    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f\": container with ID starting with 0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f not found: ID does not exist" containerID="0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f"
Sep 18 19:40:07.607129 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.607043    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f"} err="failed to get container status \"0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f\": rpc error: code = NotFound desc = could not find container \"0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f\": container with ID starting with 0a8fe8948178c264878cca81d0611d8190b5cf0aa8201514844ef78bb79c804f not found: ID does not exist"
Sep 18 19:40:07.614285 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.614260    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85f9f87b54-l6brj"]
Sep 18 19:40:07.620591 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:07.620567    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85f9f87b54-l6brj"]
Sep 18 19:40:08.927197 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:08.927155    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da143f0d-d92c-4a86-a2e7-9ab6c1212548" path="/var/lib/kubelet/pods/da143f0d-d92c-4a86-a2e7-9ab6c1212548/volumes"
Sep 18 19:40:09.492806 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:09.492769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qvw9h"
Sep 18 19:40:16.605464 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.605418    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cd7496b9d-64hzj" podUID="5b48c963-a1a2-4547-865c-550018e713d6" containerName="console" containerID="cri-o://b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a" gracePeriod=15
Sep 18 19:40:16.889947 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.889842    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:40:16.948621 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.948582    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-service-ca\") pod \"5b48c963-a1a2-4547-865c-550018e713d6\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") "
Sep 18 19:40:16.948809 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.948643    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-oauth-serving-cert\") pod \"5b48c963-a1a2-4547-865c-550018e713d6\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") "
Sep 18 19:40:16.948809 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.948668    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-console-config\") pod \"5b48c963-a1a2-4547-865c-550018e713d6\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") "
Sep 18 19:40:16.948809 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.948784    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-oauth-config\") pod \"5b48c963-a1a2-4547-865c-550018e713d6\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") "
Sep 18 19:40:16.949016 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.948834    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-trusted-ca-bundle\") pod \"5b48c963-a1a2-4547-865c-550018e713d6\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") "
Sep 18 19:40:16.949016 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.948923    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgrtc\" (UniqueName: \"kubernetes.io/projected/5b48c963-a1a2-4547-865c-550018e713d6-kube-api-access-hgrtc\") pod \"5b48c963-a1a2-4547-865c-550018e713d6\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") "
Sep 18 19:40:16.949016 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.948963    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-serving-cert\") pod \"5b48c963-a1a2-4547-865c-550018e713d6\" (UID: \"5b48c963-a1a2-4547-865c-550018e713d6\") "
Sep 18 19:40:16.949163 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.949062    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b48c963-a1a2-4547-865c-550018e713d6" (UID: "5b48c963-a1a2-4547-865c-550018e713d6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:16.949243 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.949217    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b48c963-a1a2-4547-865c-550018e713d6" (UID: "5b48c963-a1a2-4547-865c-550018e713d6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:16.949243 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.949223    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-console-config" (OuterVolumeSpecName: "console-config") pod "5b48c963-a1a2-4547-865c-550018e713d6" (UID: "5b48c963-a1a2-4547-865c-550018e713d6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:16.949343 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.949317    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-console-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:16.949343 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.949337    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-service-ca\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:16.949416 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.949351    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-oauth-serving-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:16.949416 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.949349    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5b48c963-a1a2-4547-865c-550018e713d6" (UID: "5b48c963-a1a2-4547-865c-550018e713d6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:16.951045 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.951015    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b48c963-a1a2-4547-865c-550018e713d6" (UID: "5b48c963-a1a2-4547-865c-550018e713d6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:16.951247 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.951224    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b48c963-a1a2-4547-865c-550018e713d6" (UID: "5b48c963-a1a2-4547-865c-550018e713d6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:16.951294 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:16.951275    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b48c963-a1a2-4547-865c-550018e713d6-kube-api-access-hgrtc" (OuterVolumeSpecName: "kube-api-access-hgrtc") pod "5b48c963-a1a2-4547-865c-550018e713d6" (UID: "5b48c963-a1a2-4547-865c-550018e713d6"). InnerVolumeSpecName "kube-api-access-hgrtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:17.050684 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.050645    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-oauth-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:17.050684 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.050676    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48c963-a1a2-4547-865c-550018e713d6-trusted-ca-bundle\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:17.050684 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.050685    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgrtc\" (UniqueName: \"kubernetes.io/projected/5b48c963-a1a2-4547-865c-550018e713d6-kube-api-access-hgrtc\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:17.050684 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.050695    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b48c963-a1a2-4547-865c-550018e713d6-console-serving-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:40:17.636644 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.636601    2560 generic.go:356] "Generic (PLEG): container finished" podID="5b48c963-a1a2-4547-865c-550018e713d6" containerID="b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a" exitCode=0
Sep 18 19:40:17.637066 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.636683    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd7496b9d-64hzj"
Sep 18 19:40:17.637066 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.636690    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd7496b9d-64hzj" event={"ID":"5b48c963-a1a2-4547-865c-550018e713d6","Type":"ContainerDied","Data":"b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a"}
Sep 18 19:40:17.637066 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.636776    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd7496b9d-64hzj" event={"ID":"5b48c963-a1a2-4547-865c-550018e713d6","Type":"ContainerDied","Data":"dd994f23e6e57450e80e944c17e96a2e7387007eb348930837707d33b39a46e5"}
Sep 18 19:40:17.637066 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.636799    2560 scope.go:117] "RemoveContainer" containerID="b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a"
Sep 18 19:40:17.645209 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.645193    2560 scope.go:117] "RemoveContainer" containerID="b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a"
Sep 18 19:40:17.645486 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:40:17.645462    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a\": container with ID starting with b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a not found: ID does not exist" containerID="b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a"
Sep 18 19:40:17.645537 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.645494    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a"} err="failed to get container status \"b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a\": rpc error: code = NotFound desc = could not find container \"b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a\": container with ID starting with b337508bd815ec3ef8a72e7bffc9f704c815694b61fc3162eb33451762ca448a not found: ID does not exist"
Sep 18 19:40:17.663425 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.663394    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd7496b9d-64hzj"]
Sep 18 19:40:17.666918 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:17.666894    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cd7496b9d-64hzj"]
Sep 18 19:40:18.926800 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:18.926766    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b48c963-a1a2-4547-865c-550018e713d6" path="/var/lib/kubelet/pods/5b48c963-a1a2-4547-865c-550018e713d6/volumes"
Sep 18 19:40:31.389576 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:31.389527    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:31.449381 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:31.449342    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:34.369248 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.369212    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6964f4545-j62vd"]
Sep 18 19:40:34.369620 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.369485    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b48c963-a1a2-4547-865c-550018e713d6" containerName="console"
Sep 18 19:40:34.369620 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.369497    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b48c963-a1a2-4547-865c-550018e713d6" containerName="console"
Sep 18 19:40:34.369620 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.369507    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da143f0d-d92c-4a86-a2e7-9ab6c1212548" containerName="console"
Sep 18 19:40:34.369620 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.369513    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="da143f0d-d92c-4a86-a2e7-9ab6c1212548" containerName="console"
Sep 18 19:40:34.369620 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.369560    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b48c963-a1a2-4547-865c-550018e713d6" containerName="console"
Sep 18 19:40:34.369620 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.369568    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="da143f0d-d92c-4a86-a2e7-9ab6c1212548" containerName="console"
Sep 18 19:40:34.371072 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.371056    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.384327 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.384297    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6964f4545-j62vd"]
Sep 18 19:40:34.449419 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.449386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:34.487394 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.487360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-serving-cert\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.487394 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.487393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-oauth-serving-cert\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.487581 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.487418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-config\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.487581 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.487478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-trusted-ca-bundle\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.487649 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.487583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-oauth-config\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.487649 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.487629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-service-ca\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.487710 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.487654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqfl\" (UniqueName: \"kubernetes.io/projected/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-kube-api-access-chqfl\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.550535 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.550498    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:34.588556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.588519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-serving-cert\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.588556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.588560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-oauth-serving-cert\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.588801 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.588586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-config\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.588801 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.588708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-trusted-ca-bundle\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.588801 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.588768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-oauth-config\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.588993 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.588830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-service-ca\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.588993 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.588906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqfl\" (UniqueName: \"kubernetes.io/projected/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-kube-api-access-chqfl\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.589416 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.589388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-config\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.589506 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.589430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-oauth-serving-cert\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.589587 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.589567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-trusted-ca-bundle\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.589621 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.589574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-service-ca\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.591600 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.591573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-oauth-config\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.591696 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.591614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-serving-cert\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.597945 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.597921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqfl\" (UniqueName: \"kubernetes.io/projected/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-kube-api-access-chqfl\") pod \"console-6964f4545-j62vd\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") " pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.679834 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.679792    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:34.951668 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:34.951569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6964f4545-j62vd"]
Sep 18 19:40:34.954324 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:40:34.954300    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64fa3e4_2b5b_4d1d_99b8_72ac85a11310.slice/crio-8340d096d92e8290af83d486860e14cf1e99b003495eddd8ddf5f5ef17a4cad3 WatchSource:0}: Error finding container 8340d096d92e8290af83d486860e14cf1e99b003495eddd8ddf5f5ef17a4cad3: Status 404 returned error can't find the container with id 8340d096d92e8290af83d486860e14cf1e99b003495eddd8ddf5f5ef17a4cad3
Sep 18 19:40:35.705144 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:35.705099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6964f4545-j62vd" event={"ID":"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310","Type":"ContainerStarted","Data":"77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84"}
Sep 18 19:40:35.705590 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:35.705156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6964f4545-j62vd" event={"ID":"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310","Type":"ContainerStarted","Data":"8340d096d92e8290af83d486860e14cf1e99b003495eddd8ddf5f5ef17a4cad3"}
Sep 18 19:40:35.722950 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:35.722882    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6964f4545-j62vd" podStartSLOduration=1.7228417980000001 podStartE2EDuration="1.722841798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:35.721479467 +0000 UTC m=+143.323842663" watchObservedRunningTime="2026-09-18 19:40:35.722841798 +0000 UTC m=+143.325204985"
Sep 18 19:40:37.913262 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:37.913210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:38.013723 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:38.013697    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:43.343379 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:43.343340    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:43.399799 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:43.399765    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:44.680104 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:44.680061    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:44.680489 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:44.680132    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:44.684634 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:44.684614    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:44.741489 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:44.741456    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:40:44.792692 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:44.792653    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d7874d8d-ndqpc"]
Sep 18 19:40:44.901554 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:44.901526    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:45.002934 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:45.002820    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:45.064493 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:45.064459    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:46.388944 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:46.388907    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:47.625136 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:47.625094    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:47.725656 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:47.725611    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:47.779088 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:40:47.779047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:09.812121 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:09.812010    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59d7874d8d-ndqpc" podUID="0f07a55a-618a-4244-a730-e123b75df46b" containerName="console" containerID="cri-o://0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8" gracePeriod=15
Sep 18 19:41:10.054881 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.054842    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:41:10.155186 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155154    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-console-config\") pod \"0f07a55a-618a-4244-a730-e123b75df46b\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") "
Sep 18 19:41:10.155374 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-oauth-config\") pod \"0f07a55a-618a-4244-a730-e123b75df46b\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") "
Sep 18 19:41:10.155374 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155235    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82qlg\" (UniqueName: \"kubernetes.io/projected/0f07a55a-618a-4244-a730-e123b75df46b-kube-api-access-82qlg\") pod \"0f07a55a-618a-4244-a730-e123b75df46b\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") "
Sep 18 19:41:10.155374 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155351    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-oauth-serving-cert\") pod \"0f07a55a-618a-4244-a730-e123b75df46b\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") "
Sep 18 19:41:10.155533 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155432    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-trusted-ca-bundle\") pod \"0f07a55a-618a-4244-a730-e123b75df46b\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") "
Sep 18 19:41:10.155533 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-service-ca\") pod \"0f07a55a-618a-4244-a730-e123b75df46b\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") "
Sep 18 19:41:10.155533 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-serving-cert\") pod \"0f07a55a-618a-4244-a730-e123b75df46b\" (UID: \"0f07a55a-618a-4244-a730-e123b75df46b\") "
Sep 18 19:41:10.155682 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155556    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-console-config" (OuterVolumeSpecName: "console-config") pod "0f07a55a-618a-4244-a730-e123b75df46b" (UID: "0f07a55a-618a-4244-a730-e123b75df46b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:10.155754 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155731    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-console-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:10.155803 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155765    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0f07a55a-618a-4244-a730-e123b75df46b" (UID: "0f07a55a-618a-4244-a730-e123b75df46b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:10.155886 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155833    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0f07a55a-618a-4244-a730-e123b75df46b" (UID: "0f07a55a-618a-4244-a730-e123b75df46b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:10.155952 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.155888    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-service-ca" (OuterVolumeSpecName: "service-ca") pod "0f07a55a-618a-4244-a730-e123b75df46b" (UID: "0f07a55a-618a-4244-a730-e123b75df46b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:10.157474 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.157447    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f07a55a-618a-4244-a730-e123b75df46b-kube-api-access-82qlg" (OuterVolumeSpecName: "kube-api-access-82qlg") pod "0f07a55a-618a-4244-a730-e123b75df46b" (UID: "0f07a55a-618a-4244-a730-e123b75df46b"). InnerVolumeSpecName "kube-api-access-82qlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:10.157601 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.157528    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0f07a55a-618a-4244-a730-e123b75df46b" (UID: "0f07a55a-618a-4244-a730-e123b75df46b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:10.157601 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.157548    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0f07a55a-618a-4244-a730-e123b75df46b" (UID: "0f07a55a-618a-4244-a730-e123b75df46b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:10.256302 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.256259    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-oauth-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:10.256302 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.256292    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qlg\" (UniqueName: \"kubernetes.io/projected/0f07a55a-618a-4244-a730-e123b75df46b-kube-api-access-82qlg\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:10.256302 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.256304    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-oauth-serving-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:10.256302 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.256313    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-trusted-ca-bundle\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:10.256553 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.256322    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f07a55a-618a-4244-a730-e123b75df46b-service-ca\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:10.256553 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.256330    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f07a55a-618a-4244-a730-e123b75df46b-console-serving-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:10.839785 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.839745    2560 generic.go:356] "Generic (PLEG): container finished" podID="0f07a55a-618a-4244-a730-e123b75df46b" containerID="0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8" exitCode=0
Sep 18 19:41:10.840186 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.839793    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d7874d8d-ndqpc"
Sep 18 19:41:10.840186 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.839795    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d7874d8d-ndqpc" event={"ID":"0f07a55a-618a-4244-a730-e123b75df46b","Type":"ContainerDied","Data":"0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8"}
Sep 18 19:41:10.840186 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.839963    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d7874d8d-ndqpc" event={"ID":"0f07a55a-618a-4244-a730-e123b75df46b","Type":"ContainerDied","Data":"ae22fb8049eb8fb711a0c88ee9f76d41ead200c033f84d2a90b580319f80bccb"}
Sep 18 19:41:10.840186 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.840004    2560 scope.go:117] "RemoveContainer" containerID="0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8"
Sep 18 19:41:10.848380 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.848360    2560 scope.go:117] "RemoveContainer" containerID="0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8"
Sep 18 19:41:10.848702 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:41:10.848683    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8\": container with ID starting with 0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8 not found: ID does not exist" containerID="0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8"
Sep 18 19:41:10.848748 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.848711    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8"} err="failed to get container status \"0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8\": rpc error: code = NotFound desc = could not find container \"0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8\": container with ID starting with 0d74838bc49119d48c2f772859e2a73539e058e099bb7ccd7b6c61a960b59ba8 not found: ID does not exist"
Sep 18 19:41:10.859192 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.859161    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d7874d8d-ndqpc"]
Sep 18 19:41:10.862192 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.862166    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59d7874d8d-ndqpc"]
Sep 18 19:41:10.927113 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:10.927070    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f07a55a-618a-4244-a730-e123b75df46b" path="/var/lib/kubelet/pods/0f07a55a-618a-4244-a730-e123b75df46b/volumes"
Sep 18 19:41:16.846473 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:16.846434    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:36.779436 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:36.779401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:36.878785 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:36.878746    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:36.936567 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:36.936528    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:37.722541 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.722503    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"]
Sep 18 19:41:37.722774 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.722762    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f07a55a-618a-4244-a730-e123b75df46b" containerName="console"
Sep 18 19:41:37.722820 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.722775    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f07a55a-618a-4244-a730-e123b75df46b" containerName="console"
Sep 18 19:41:37.722875 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.722830    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f07a55a-618a-4244-a730-e123b75df46b" containerName="console"
Sep 18 19:41:37.724731 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.724714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.727220 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.727201    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:41:37.727339 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.727277    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t2r8s\""
Sep 18 19:41:37.728537 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.728521    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:41:37.736384 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.736356    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"]
Sep 18 19:41:37.864344 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.864303    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.864344 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.864349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgrc\" (UniqueName: \"kubernetes.io/projected/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-kube-api-access-gqgrc\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.864735 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.864425    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.965343 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.965309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.965500 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.965366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.965500 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.965391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgrc\" (UniqueName: \"kubernetes.io/projected/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-kube-api-access-gqgrc\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.965722 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.965702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.965757 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.965733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:37.973202 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:37.973134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgrc\" (UniqueName: \"kubernetes.io/projected/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-kube-api-access-gqgrc\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:38.033057 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:38.033018    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:38.301802 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:38.301718    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"]
Sep 18 19:41:38.306065 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:41:38.306040    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b82a30_548c_4ca2_8c13_f67d768bfa6f.slice/crio-16c0209020e9c5fe0fd76c2df08985442a377d0817b431edec7d99d77f1a8096 WatchSource:0}: Error finding container 16c0209020e9c5fe0fd76c2df08985442a377d0817b431edec7d99d77f1a8096: Status 404 returned error can't find the container with id 16c0209020e9c5fe0fd76c2df08985442a377d0817b431edec7d99d77f1a8096
Sep 18 19:41:38.941735 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:38.941700    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h" event={"ID":"f0b82a30-548c-4ca2-8c13-f67d768bfa6f","Type":"ContainerStarted","Data":"16c0209020e9c5fe0fd76c2df08985442a377d0817b431edec7d99d77f1a8096"}
Sep 18 19:41:43.961373 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:43.961334    2560 generic.go:356] "Generic (PLEG): container finished" podID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerID="eebef4a5765948526c5ee060b1ae98aeba86d777df5833f6fcc3faacbb16ac2e" exitCode=0
Sep 18 19:41:43.961780 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:43.961420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h" event={"ID":"f0b82a30-548c-4ca2-8c13-f67d768bfa6f","Type":"ContainerDied","Data":"eebef4a5765948526c5ee060b1ae98aeba86d777df5833f6fcc3faacbb16ac2e"}
Sep 18 19:41:46.974274 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:46.974239    2560 generic.go:356] "Generic (PLEG): container finished" podID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerID="eed390ff262851e94ff0a2a5893a9e39fe17de481daa388af4b0a5b5e53309ba" exitCode=0
Sep 18 19:41:46.974652 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:46.974285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h" event={"ID":"f0b82a30-548c-4ca2-8c13-f67d768bfa6f","Type":"ContainerDied","Data":"eed390ff262851e94ff0a2a5893a9e39fe17de481daa388af4b0a5b5e53309ba"}
Sep 18 19:41:55.012498 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:55.012465    2560 generic.go:356] "Generic (PLEG): container finished" podID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerID="a3c82e996d48b7663e631581e12ba18ce780049d003370caf89589c3c2584dda" exitCode=0
Sep 18 19:41:55.012945 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:55.012523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h" event={"ID":"f0b82a30-548c-4ca2-8c13-f67d768bfa6f","Type":"ContainerDied","Data":"a3c82e996d48b7663e631581e12ba18ce780049d003370caf89589c3c2584dda"}
Sep 18 19:41:56.134385 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.134360    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:56.224482 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.224447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqgrc\" (UniqueName: \"kubernetes.io/projected/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-kube-api-access-gqgrc\") pod \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") "
Sep 18 19:41:56.224644 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.224517    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-bundle\") pod \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") "
Sep 18 19:41:56.224644 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.224555    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-util\") pod \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\" (UID: \"f0b82a30-548c-4ca2-8c13-f67d768bfa6f\") "
Sep 18 19:41:56.225231 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.225199    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-bundle" (OuterVolumeSpecName: "bundle") pod "f0b82a30-548c-4ca2-8c13-f67d768bfa6f" (UID: "f0b82a30-548c-4ca2-8c13-f67d768bfa6f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:56.226705 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.226675    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-kube-api-access-gqgrc" (OuterVolumeSpecName: "kube-api-access-gqgrc") pod "f0b82a30-548c-4ca2-8c13-f67d768bfa6f" (UID: "f0b82a30-548c-4ca2-8c13-f67d768bfa6f"). InnerVolumeSpecName "kube-api-access-gqgrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:56.229074 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.229055    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-util" (OuterVolumeSpecName: "util") pod "f0b82a30-548c-4ca2-8c13-f67d768bfa6f" (UID: "f0b82a30-548c-4ca2-8c13-f67d768bfa6f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:56.325281 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.325194    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgrc\" (UniqueName: \"kubernetes.io/projected/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-kube-api-access-gqgrc\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:56.325281 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.325225    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-bundle\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:56.325281 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:56.325234    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0b82a30-548c-4ca2-8c13-f67d768bfa6f-util\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:41:57.021291 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:57.021258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h" event={"ID":"f0b82a30-548c-4ca2-8c13-f67d768bfa6f","Type":"ContainerDied","Data":"16c0209020e9c5fe0fd76c2df08985442a377d0817b431edec7d99d77f1a8096"}
Sep 18 19:41:57.021450 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:57.021299    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahqq9h"
Sep 18 19:41:57.021450 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:57.021304    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c0209020e9c5fe0fd76c2df08985442a377d0817b431edec7d99d77f1a8096"
Sep 18 19:41:58.778268 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:58.778234    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:58.864503 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:58.864474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:59.759540 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.759506    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"]
Sep 18 19:41:59.759793 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.759781    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerName="pull"
Sep 18 19:41:59.759841 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.759796    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerName="pull"
Sep 18 19:41:59.759841 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.759806    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerName="extract"
Sep 18 19:41:59.759841 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.759812    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerName="extract"
Sep 18 19:41:59.759841 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.759821    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerName="util"
Sep 18 19:41:59.759841 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.759826    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerName="util"
Sep 18 19:41:59.760004 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.759894    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b82a30-548c-4ca2-8c13-f67d768bfa6f" containerName="extract"
Sep 18 19:41:59.763173 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.763154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:41:59.766152 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.766129    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-d2nzn\""
Sep 18 19:41:59.766273 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.766169    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 19:41:59.766273 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.766180    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 19:41:59.766273 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.766238    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 19:41:59.774504 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.774479    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"]
Sep 18 19:41:59.952002 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.951947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a464aa71-5f7c-4200-bab5-d6e82df5f67b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9pdn4\" (UID: \"a464aa71-5f7c-4200-bab5-d6e82df5f67b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:41:59.952398 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:41:59.952061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bll\" (UniqueName: \"kubernetes.io/projected/a464aa71-5f7c-4200-bab5-d6e82df5f67b-kube-api-access-v4bll\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9pdn4\" (UID: \"a464aa71-5f7c-4200-bab5-d6e82df5f67b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:42:00.053450 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:00.053349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a464aa71-5f7c-4200-bab5-d6e82df5f67b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9pdn4\" (UID: \"a464aa71-5f7c-4200-bab5-d6e82df5f67b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:42:00.053450 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:00.053430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bll\" (UniqueName: \"kubernetes.io/projected/a464aa71-5f7c-4200-bab5-d6e82df5f67b-kube-api-access-v4bll\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9pdn4\" (UID: \"a464aa71-5f7c-4200-bab5-d6e82df5f67b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:42:00.055669 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:00.055647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a464aa71-5f7c-4200-bab5-d6e82df5f67b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9pdn4\" (UID: \"a464aa71-5f7c-4200-bab5-d6e82df5f67b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:42:00.062061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:00.062036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bll\" (UniqueName: \"kubernetes.io/projected/a464aa71-5f7c-4200-bab5-d6e82df5f67b-kube-api-access-v4bll\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9pdn4\" (UID: \"a464aa71-5f7c-4200-bab5-d6e82df5f67b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:42:00.073017 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:00.072989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:42:00.351025 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:00.350920    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"]
Sep 18 19:42:00.354678 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:42:00.354650    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda464aa71_5f7c_4200_bab5_d6e82df5f67b.slice/crio-fecf7eb618d4ee90566c47affdf1defd2a1f9a83fbd283c942fcd3a109f33c25 WatchSource:0}: Error finding container fecf7eb618d4ee90566c47affdf1defd2a1f9a83fbd283c942fcd3a109f33c25: Status 404 returned error can't find the container with id fecf7eb618d4ee90566c47affdf1defd2a1f9a83fbd283c942fcd3a109f33c25
Sep 18 19:42:01.035965 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:01.035932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4" event={"ID":"a464aa71-5f7c-4200-bab5-d6e82df5f67b","Type":"ContainerStarted","Data":"fecf7eb618d4ee90566c47affdf1defd2a1f9a83fbd283c942fcd3a109f33c25"}
Sep 18 19:42:02.400682 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:02.400643    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:02.501481 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:02.501438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:03.863309 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:03.863219    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.049904 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.049868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4" event={"ID":"a464aa71-5f7c-4200-bab5-d6e82df5f67b","Type":"ContainerStarted","Data":"8feb829ddfa1ed8cd1e8eac77cf504ffc5600acdd710de79246501e2932f9b8b"}
Sep 18 19:42:04.050069 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.049956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:42:04.070898 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.070733    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4" podStartSLOduration=1.999885409 podStartE2EDuration="5.070713708s" totalImagesPullingTime="3.070828299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:04.069454362 +0000 UTC m=+231.671817545" watchObservedRunningTime="2026-09-18 19:42:04.070713708 +0000 UTC m=+231.673076890"
Sep 18 19:42:04.137167 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.137060    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.140272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.140245    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-r8jrc"]
Sep 18 19:42:04.143533 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.143516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.146274 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.146080    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 19:42:04.146588 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.146568    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-n6mxn\""
Sep 18 19:42:04.149865 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.146674    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 19:42:04.153907 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.153873    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-r8jrc"]
Sep 18 19:42:04.288821 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.288784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-cabundle0\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.289061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.288861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7l6t\" (UniqueName: \"kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-kube-api-access-m7l6t\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.289061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.288975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-certificates\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.340699 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.340664    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"]
Sep 18 19:42:04.343633 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.343589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.344534 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.344513    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.347157 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.347139    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 19:42:04.355631 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.355610    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"]
Sep 18 19:42:04.390116 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.390035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-cabundle0\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.390116 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.390095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7l6t\" (UniqueName: \"kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-kube-api-access-m7l6t\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.390344 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.390136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-certificates\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.390344 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:42:04.390253    2560 secret.go:281] references non-existent secret key: ca.crt
Sep 18 19:42:04.390344 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:42:04.390271    2560 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 19:42:04.390344 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:42:04.390282    2560 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-r8jrc: references non-existent secret key: ca.crt
Sep 18 19:42:04.390519 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:42:04.390360    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-certificates podName:1a618d3a-ef6f-4f9f-9637-1cd620c897bd nodeName:}" failed. No retries permitted until 2026-09-18 19:42:04.890337966 +0000 UTC m=+232.492701143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-certificates") pod "keda-operator-579c8958f7-r8jrc" (UID: "1a618d3a-ef6f-4f9f-9637-1cd620c897bd") : references non-existent secret key: ca.crt
Sep 18 19:42:04.390807 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.390785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-cabundle0\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.400418 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.400394    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7l6t\" (UniqueName: \"kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-kube-api-access-m7l6t\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.436763 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.436725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.491189 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.491153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmk2s\" (UniqueName: \"kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-kube-api-access-lmk2s\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.491354 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.491226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.491354 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.491251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.494423 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.494391    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.592612 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.592569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmk2s\" (UniqueName: \"kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-kube-api-access-lmk2s\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.592800 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.592673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.592800 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.592709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.592904 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:42:04.592806    2560 secret.go:281] references non-existent secret key: tls.crt
Sep 18 19:42:04.592904 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:42:04.592829    2560 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 18 19:42:04.592904 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:42:04.592865    2560 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg: references non-existent secret key: tls.crt
Sep 18 19:42:04.593024 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:42:04.592934    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-certificates podName:bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c nodeName:}" failed. No retries permitted until 2026-09-18 19:42:05.092906066 +0000 UTC m=+232.695269235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-certificates") pod "keda-metrics-apiserver-77857b7b98-rr8xg" (UID: "bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c") : references non-existent secret key: tls.crt
Sep 18 19:42:04.593203 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.593183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.603603 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.603581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmk2s\" (UniqueName: \"kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-kube-api-access-lmk2s\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:04.743872 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.743820    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.744263 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.744237    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ddznm"]
Sep 18 19:42:04.747396 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.747380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:04.749908 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.749888    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 19:42:04.757743 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.757721    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ddznm"]
Sep 18 19:42:04.840807 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.840772    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.895100 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.895060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c57a786-91b6-48d1-8b50-9c2c641cc2e9-certificates\") pod \"keda-admission-6c4d864b86-ddznm\" (UID: \"8c57a786-91b6-48d1-8b50-9c2c641cc2e9\") " pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:04.895566 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.895129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-certificates\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.895566 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.895228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpgk\" (UniqueName: \"kubernetes.io/projected/8c57a786-91b6-48d1-8b50-9c2c641cc2e9-kube-api-access-dvpgk\") pod \"keda-admission-6c4d864b86-ddznm\" (UID: \"8c57a786-91b6-48d1-8b50-9c2c641cc2e9\") " pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:04.898208 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.898177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1a618d3a-ef6f-4f9f-9637-1cd620c897bd-certificates\") pod \"keda-operator-579c8958f7-r8jrc\" (UID: \"1a618d3a-ef6f-4f9f-9637-1cd620c897bd\") " pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:04.958823 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.958785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:04.995989 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.995885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c57a786-91b6-48d1-8b50-9c2c641cc2e9-certificates\") pod \"keda-admission-6c4d864b86-ddznm\" (UID: \"8c57a786-91b6-48d1-8b50-9c2c641cc2e9\") " pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:04.996165 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.996024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpgk\" (UniqueName: \"kubernetes.io/projected/8c57a786-91b6-48d1-8b50-9c2c641cc2e9-kube-api-access-dvpgk\") pod \"keda-admission-6c4d864b86-ddznm\" (UID: \"8c57a786-91b6-48d1-8b50-9c2c641cc2e9\") " pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:04.999195 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:04.999155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c57a786-91b6-48d1-8b50-9c2c641cc2e9-certificates\") pod \"keda-admission-6c4d864b86-ddznm\" (UID: \"8c57a786-91b6-48d1-8b50-9c2c641cc2e9\") " pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:05.005824 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.005788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpgk\" (UniqueName: \"kubernetes.io/projected/8c57a786-91b6-48d1-8b50-9c2c641cc2e9-kube-api-access-dvpgk\") pod \"keda-admission-6c4d864b86-ddznm\" (UID: \"8c57a786-91b6-48d1-8b50-9c2c641cc2e9\") " pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:05.057500 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.057463    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:05.060008 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.059976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:05.069661 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.069621    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:05.096862 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.096745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:05.101194 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.101096    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rr8xg\" (UID: \"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:05.159348 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.159317    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:05.254425 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.254296    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:05.341903 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.341873    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ddznm"]
Sep 18 19:42:05.344313 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:42:05.344287    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c57a786_91b6_48d1_8b50_9c2c641cc2e9.slice/crio-52c772f06a14a18103e56995f522a6b48879d516c0368d32c8e7aeb92c995439 WatchSource:0}: Error finding container 52c772f06a14a18103e56995f522a6b48879d516c0368d32c8e7aeb92c995439: Status 404 returned error can't find the container with id 52c772f06a14a18103e56995f522a6b48879d516c0368d32c8e7aeb92c995439
Sep 18 19:42:05.347022 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.346980    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:05.347459 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.347437    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-r8jrc"]
Sep 18 19:42:05.349758 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:42:05.349732    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a618d3a_ef6f_4f9f_9637_1cd620c897bd.slice/crio-54a3411397a841e831053b732f46f400595724dcedcf6917a2c249d048962a36 WatchSource:0}: Error finding container 54a3411397a841e831053b732f46f400595724dcedcf6917a2c249d048962a36: Status 404 returned error can't find the container with id 54a3411397a841e831053b732f46f400595724dcedcf6917a2c249d048962a36
Sep 18 19:42:05.406924 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.406895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:05.533000 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:05.532813    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"]
Sep 18 19:42:05.535386 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:42:05.535361    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa6b2f8_4179_4b45_a4c1_f7d3d8d1764c.slice/crio-210b3609204b2be9d88698b1aba182e2f6b96e5469bb7c9ec97de4d864ba28d9 WatchSource:0}: Error finding container 210b3609204b2be9d88698b1aba182e2f6b96e5469bb7c9ec97de4d864ba28d9: Status 404 returned error can't find the container with id 210b3609204b2be9d88698b1aba182e2f6b96e5469bb7c9ec97de4d864ba28d9
Sep 18 19:42:06.059403 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:06.059362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ddznm" event={"ID":"8c57a786-91b6-48d1-8b50-9c2c641cc2e9","Type":"ContainerStarted","Data":"52c772f06a14a18103e56995f522a6b48879d516c0368d32c8e7aeb92c995439"}
Sep 18 19:42:06.060531 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:06.060504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-r8jrc" event={"ID":"1a618d3a-ef6f-4f9f-9637-1cd620c897bd","Type":"ContainerStarted","Data":"54a3411397a841e831053b732f46f400595724dcedcf6917a2c249d048962a36"}
Sep 18 19:42:06.064183 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:06.064149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg" event={"ID":"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c","Type":"ContainerStarted","Data":"210b3609204b2be9d88698b1aba182e2f6b96e5469bb7c9ec97de4d864ba28d9"}
Sep 18 19:42:08.076380 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:08.076340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ddznm" event={"ID":"8c57a786-91b6-48d1-8b50-9c2c641cc2e9","Type":"ContainerStarted","Data":"762d6d9504b44ac459db52ee921126e35e9bed6ff917e8af56da726e6816e6ac"}
Sep 18 19:42:08.076762 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:08.076502    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:08.093327 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:08.093283    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-ddznm" podStartSLOduration=2.139640257 podStartE2EDuration="4.093267478s" totalImagesPullingTime="1.953627221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:08.092130605 +0000 UTC m=+235.694493786" watchObservedRunningTime="2026-09-18 19:42:08.093267478 +0000 UTC m=+235.695630659"
Sep 18 19:42:10.088506 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:10.088466    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-r8jrc" event={"ID":"1a618d3a-ef6f-4f9f-9637-1cd620c897bd","Type":"ContainerStarted","Data":"3817dc785a4b4d69ae73d0a84804c4b82975341c5f1bacb881dfbdb526eff685"}
Sep 18 19:42:10.089086 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:10.088622    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:10.106391 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:10.106341    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-r8jrc" podStartSLOduration=1.575975519 podStartE2EDuration="6.106324914s" totalImagesPullingTime="4.530349395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:10.104029056 +0000 UTC m=+237.706392230" watchObservedRunningTime="2026-09-18 19:42:10.106324914 +0000 UTC m=+237.708688095"
Sep 18 19:42:15.113948 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:15.113905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg" event={"ID":"bfa6b2f8-4179-4b45-a4c1-f7d3d8d1764c","Type":"ContainerStarted","Data":"6d0ae68c48749159d60c51a42694fa6cab42579d2e94f2c90c06b2ab9c4f601d"}
Sep 18 19:42:15.114397 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:15.114044    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:15.132178 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:15.132114    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg" podStartSLOduration=2.260288963 podStartE2EDuration="11.13209609s" totalImagesPullingTime="8.871807127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:15.130706431 +0000 UTC m=+242.733069642" watchObservedRunningTime="2026-09-18 19:42:15.13209609 +0000 UTC m=+242.734459271"
Sep 18 19:42:25.056880 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:25.056827    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9pdn4"
Sep 18 19:42:26.123648 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:26.123618    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rr8xg"
Sep 18 19:42:29.083902 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:29.083865    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-ddznm"
Sep 18 19:42:31.095826 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:31.095793    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-r8jrc"
Sep 18 19:42:57.239697 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:57.239579    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:58.624389 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:42:58.624356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:04.372584 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:04.372544    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:04.475909 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:04.475871    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:12.721609 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:12.721583    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:43:21.052167 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.052131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.114372 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.114338    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-d2ls5"]
Sep 18 19:43:21.117518 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.117493    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:21.120050 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.120025    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 18 19:43:21.120199 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.120075    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-fv9jw\""
Sep 18 19:43:21.121046 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.121028    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:43:21.121112 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.121035    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:43:21.128152 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.128127    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"]
Sep 18 19:43:21.133239 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.133210    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-d2ls5"]
Sep 18 19:43:21.133574 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.133549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.133750 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.133563    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:21.137069 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.137046    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-pjfk4\""
Sep 18 19:43:21.138119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.137432    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 19:43:21.143651 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.143628    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"]
Sep 18 19:43:21.210532 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.210494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcbf5392-7320-4796-b1df-099b7b688384-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-2kd9d\" (UID: \"dcbf5392-7320-4796-b1df-099b7b688384\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:21.210702 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.210546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gz2\" (UniqueName: \"kubernetes.io/projected/dcbf5392-7320-4796-b1df-099b7b688384-kube-api-access-26gz2\") pod \"llmisvc-controller-manager-645f9c8fd6-2kd9d\" (UID: \"dcbf5392-7320-4796-b1df-099b7b688384\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:21.210702 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.210564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86g7\" (UniqueName: \"kubernetes.io/projected/f17f8c7b-46f0-432c-a8cb-281446c61640-kube-api-access-d86g7\") pod \"kserve-controller-manager-b9b5d54f8-d2ls5\" (UID: \"f17f8c7b-46f0-432c-a8cb-281446c61640\") " pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:21.210702 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.210682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f17f8c7b-46f0-432c-a8cb-281446c61640-cert\") pod \"kserve-controller-manager-b9b5d54f8-d2ls5\" (UID: \"f17f8c7b-46f0-432c-a8cb-281446c61640\") " pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:21.267767 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.267721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.311660 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.311573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcbf5392-7320-4796-b1df-099b7b688384-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-2kd9d\" (UID: \"dcbf5392-7320-4796-b1df-099b7b688384\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:21.311660 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.311622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gz2\" (UniqueName: \"kubernetes.io/projected/dcbf5392-7320-4796-b1df-099b7b688384-kube-api-access-26gz2\") pod \"llmisvc-controller-manager-645f9c8fd6-2kd9d\" (UID: \"dcbf5392-7320-4796-b1df-099b7b688384\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:21.311660 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.311641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86g7\" (UniqueName: \"kubernetes.io/projected/f17f8c7b-46f0-432c-a8cb-281446c61640-kube-api-access-d86g7\") pod \"kserve-controller-manager-b9b5d54f8-d2ls5\" (UID: \"f17f8c7b-46f0-432c-a8cb-281446c61640\") " pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:21.311914 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.311711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f17f8c7b-46f0-432c-a8cb-281446c61640-cert\") pod \"kserve-controller-manager-b9b5d54f8-d2ls5\" (UID: \"f17f8c7b-46f0-432c-a8cb-281446c61640\") " pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:21.311914 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:43:21.311735    2560 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 19:43:21.311914 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:43:21.311821    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbf5392-7320-4796-b1df-099b7b688384-cert podName:dcbf5392-7320-4796-b1df-099b7b688384 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:21.811799489 +0000 UTC m=+309.414162674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dcbf5392-7320-4796-b1df-099b7b688384-cert") pod "llmisvc-controller-manager-645f9c8fd6-2kd9d" (UID: "dcbf5392-7320-4796-b1df-099b7b688384") : secret "llmisvc-webhook-server-cert" not found
Sep 18 19:43:21.314127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.314106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f17f8c7b-46f0-432c-a8cb-281446c61640-cert\") pod \"kserve-controller-manager-b9b5d54f8-d2ls5\" (UID: \"f17f8c7b-46f0-432c-a8cb-281446c61640\") " pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:21.320650 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.320621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86g7\" (UniqueName: \"kubernetes.io/projected/f17f8c7b-46f0-432c-a8cb-281446c61640-kube-api-access-d86g7\") pod \"kserve-controller-manager-b9b5d54f8-d2ls5\" (UID: \"f17f8c7b-46f0-432c-a8cb-281446c61640\") " pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:21.321149 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.321133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gz2\" (UniqueName: \"kubernetes.io/projected/dcbf5392-7320-4796-b1df-099b7b688384-kube-api-access-26gz2\") pod \"llmisvc-controller-manager-645f9c8fd6-2kd9d\" (UID: \"dcbf5392-7320-4796-b1df-099b7b688384\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:21.368528 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.368483    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.425368 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.425328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.430208 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.430187    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:21.704018 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.703994    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-d2ls5"]
Sep 18 19:43:21.706624 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:43:21.706602    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17f8c7b_46f0_432c_a8cb_281446c61640.slice/crio-72b6f3a46343cc1bfc201ca672fb46ed7e6491cf8d6a7586d9246d07c6eb9dc6 WatchSource:0}: Error finding container 72b6f3a46343cc1bfc201ca672fb46ed7e6491cf8d6a7586d9246d07c6eb9dc6: Status 404 returned error can't find the container with id 72b6f3a46343cc1bfc201ca672fb46ed7e6491cf8d6a7586d9246d07c6eb9dc6
Sep 18 19:43:21.707910 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.707893    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:43:21.817076 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.817027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcbf5392-7320-4796-b1df-099b7b688384-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-2kd9d\" (UID: \"dcbf5392-7320-4796-b1df-099b7b688384\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:21.819435 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.819414    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcbf5392-7320-4796-b1df-099b7b688384-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-2kd9d\" (UID: \"dcbf5392-7320-4796-b1df-099b7b688384\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:21.978029 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:21.977946    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:22.045927 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:22.045889    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:22.073924 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:22.073888    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:22.318258 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:22.318216    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"]
Sep 18 19:43:22.321472 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:43:22.321427    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddcbf5392_7320_4796_b1df_099b7b688384.slice/crio-f93c7c56f5aa66137d2aab4e268f34210da0f6e583ead493fed6f97915b2fe02 WatchSource:0}: Error finding container f93c7c56f5aa66137d2aab4e268f34210da0f6e583ead493fed6f97915b2fe02: Status 404 returned error can't find the container with id f93c7c56f5aa66137d2aab4e268f34210da0f6e583ead493fed6f97915b2fe02
Sep 18 19:43:22.396359 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:22.396269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"f93c7c56f5aa66137d2aab4e268f34210da0f6e583ead493fed6f97915b2fe02"}
Sep 18 19:43:22.397646 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:22.397611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5" event={"ID":"f17f8c7b-46f0-432c-a8cb-281446c61640","Type":"ContainerStarted","Data":"72b6f3a46343cc1bfc201ca672fb46ed7e6491cf8d6a7586d9246d07c6eb9dc6"}
Sep 18 19:43:25.414223 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:25.414190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"96bda9a17582683e862ad1f996790a0428bf28b5ecc643264f057674cde85646"}
Sep 18 19:43:25.414648 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:25.414297    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:25.415773 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:25.415751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5" event={"ID":"f17f8c7b-46f0-432c-a8cb-281446c61640","Type":"ContainerStarted","Data":"cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c"}
Sep 18 19:43:25.415903 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:25.415876    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:43:25.429453 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:25.429394    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podStartSLOduration=1.681302823 podStartE2EDuration="4.429373429s" totalImagesPullingTime="2.748070606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:25.429030855 +0000 UTC m=+313.031394038" watchObservedRunningTime="2026-09-18 19:43:25.429373429 +0000 UTC m=+313.031736612"
Sep 18 19:43:25.450101 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:25.450053    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5" podStartSLOduration=1.089595663 podStartE2EDuration="4.450035904s" totalImagesPullingTime="3.360440241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:25.44936278 +0000 UTC m=+313.051725963" watchObservedRunningTime="2026-09-18 19:43:25.450035904 +0000 UTC m=+313.052399088"
Sep 18 19:43:45.863374 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:45.863336    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:45.928296 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:45.928260    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:56.421884 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:56.421834    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:43:56.424868 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:43:56.424821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:44:03.174757 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.174718    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-d2ls5"]
Sep 18 19:44:03.175200 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.174940    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5" podUID="f17f8c7b-46f0-432c-a8cb-281446c61640" containerName="manager" containerID="cri-o://cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c" gracePeriod=10
Sep 18 19:44:03.196954 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.196916    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-cgnvb"]
Sep 18 19:44:03.200024 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.200008    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:03.208409 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.208380    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-cgnvb"]
Sep 18 19:44:03.368118 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.368086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrh8g\" (UniqueName: \"kubernetes.io/projected/2020bd6f-aa3d-4420-903c-0171ef1672cf-kube-api-access-qrh8g\") pod \"kserve-controller-manager-b9b5d54f8-cgnvb\" (UID: \"2020bd6f-aa3d-4420-903c-0171ef1672cf\") " pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:03.368118 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.368121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2020bd6f-aa3d-4420-903c-0171ef1672cf-cert\") pod \"kserve-controller-manager-b9b5d54f8-cgnvb\" (UID: \"2020bd6f-aa3d-4420-903c-0171ef1672cf\") " pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:03.422366 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.422332    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:44:03.469514 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.469410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrh8g\" (UniqueName: \"kubernetes.io/projected/2020bd6f-aa3d-4420-903c-0171ef1672cf-kube-api-access-qrh8g\") pod \"kserve-controller-manager-b9b5d54f8-cgnvb\" (UID: \"2020bd6f-aa3d-4420-903c-0171ef1672cf\") " pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:03.469514 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.469452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2020bd6f-aa3d-4420-903c-0171ef1672cf-cert\") pod \"kserve-controller-manager-b9b5d54f8-cgnvb\" (UID: \"2020bd6f-aa3d-4420-903c-0171ef1672cf\") " pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:03.471975 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.471948    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2020bd6f-aa3d-4420-903c-0171ef1672cf-cert\") pod \"kserve-controller-manager-b9b5d54f8-cgnvb\" (UID: \"2020bd6f-aa3d-4420-903c-0171ef1672cf\") " pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:03.478068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.478044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrh8g\" (UniqueName: \"kubernetes.io/projected/2020bd6f-aa3d-4420-903c-0171ef1672cf-kube-api-access-qrh8g\") pod \"kserve-controller-manager-b9b5d54f8-cgnvb\" (UID: \"2020bd6f-aa3d-4420-903c-0171ef1672cf\") " pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:03.552516 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.552475    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:03.570480 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.570452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d86g7\" (UniqueName: \"kubernetes.io/projected/f17f8c7b-46f0-432c-a8cb-281446c61640-kube-api-access-d86g7\") pod \"f17f8c7b-46f0-432c-a8cb-281446c61640\" (UID: \"f17f8c7b-46f0-432c-a8cb-281446c61640\") "
Sep 18 19:44:03.570606 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.570531    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f17f8c7b-46f0-432c-a8cb-281446c61640-cert\") pod \"f17f8c7b-46f0-432c-a8cb-281446c61640\" (UID: \"f17f8c7b-46f0-432c-a8cb-281446c61640\") "
Sep 18 19:44:03.572739 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.572709    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17f8c7b-46f0-432c-a8cb-281446c61640-cert" (OuterVolumeSpecName: "cert") pod "f17f8c7b-46f0-432c-a8cb-281446c61640" (UID: "f17f8c7b-46f0-432c-a8cb-281446c61640"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:03.572884 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.572778    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17f8c7b-46f0-432c-a8cb-281446c61640-kube-api-access-d86g7" (OuterVolumeSpecName: "kube-api-access-d86g7") pod "f17f8c7b-46f0-432c-a8cb-281446c61640" (UID: "f17f8c7b-46f0-432c-a8cb-281446c61640"). InnerVolumeSpecName "kube-api-access-d86g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:03.581096 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.581062    2560 generic.go:356] "Generic (PLEG): container finished" podID="f17f8c7b-46f0-432c-a8cb-281446c61640" containerID="cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c" exitCode=0
Sep 18 19:44:03.581257 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.581115    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5"
Sep 18 19:44:03.581257 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.581143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5" event={"ID":"f17f8c7b-46f0-432c-a8cb-281446c61640","Type":"ContainerDied","Data":"cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c"}
Sep 18 19:44:03.581257 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.581215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-d2ls5" event={"ID":"f17f8c7b-46f0-432c-a8cb-281446c61640","Type":"ContainerDied","Data":"72b6f3a46343cc1bfc201ca672fb46ed7e6491cf8d6a7586d9246d07c6eb9dc6"}
Sep 18 19:44:03.581257 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.581247    2560 scope.go:117] "RemoveContainer" containerID="cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c"
Sep 18 19:44:03.589205 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.589143    2560 scope.go:117] "RemoveContainer" containerID="cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c"
Sep 18 19:44:03.590714 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:44:03.590687    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c\": container with ID starting with cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c not found: ID does not exist" containerID="cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c"
Sep 18 19:44:03.590818 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.590726    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c"} err="failed to get container status \"cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c\": rpc error: code = NotFound desc = could not find container \"cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c\": container with ID starting with cbeec91558c12f4d5d839e32c3f5530471541be8fa94aaf7d6c6f010c81cf07c not found: ID does not exist"
Sep 18 19:44:03.604569 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.604542    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-d2ls5"]
Sep 18 19:44:03.609122 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.609091    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-d2ls5"]
Sep 18 19:44:03.672032 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.672003    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86g7\" (UniqueName: \"kubernetes.io/projected/f17f8c7b-46f0-432c-a8cb-281446c61640-kube-api-access-d86g7\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:44:03.672032 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.672031    2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f17f8c7b-46f0-432c-a8cb-281446c61640-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:44:03.822983 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:03.822963    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b9b5d54f8-cgnvb"]
Sep 18 19:44:03.825515 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:44:03.825491    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2020bd6f_aa3d_4420_903c_0171ef1672cf.slice/crio-65d18113bef1918a3c8eccf8298f68c6acbfafa04bde42c702ddc01af8460c01 WatchSource:0}: Error finding container 65d18113bef1918a3c8eccf8298f68c6acbfafa04bde42c702ddc01af8460c01: Status 404 returned error can't find the container with id 65d18113bef1918a3c8eccf8298f68c6acbfafa04bde42c702ddc01af8460c01
Sep 18 19:44:04.587618 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:04.587523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb" event={"ID":"2020bd6f-aa3d-4420-903c-0171ef1672cf","Type":"ContainerStarted","Data":"5d78173c23c037f3f57bc31c37316c91c7a4fdcf2f298908e36b441c6218771b"}
Sep 18 19:44:04.587618 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:04.587571    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb" event={"ID":"2020bd6f-aa3d-4420-903c-0171ef1672cf","Type":"ContainerStarted","Data":"65d18113bef1918a3c8eccf8298f68c6acbfafa04bde42c702ddc01af8460c01"}
Sep 18 19:44:04.588060 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:04.587646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:04.605501 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:04.605439    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb" podStartSLOduration=1.273404951 podStartE2EDuration="1.605418292s" totalImagesPullingTime="332.013341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:04.603958355 +0000 UTC m=+352.206321539" watchObservedRunningTime="2026-09-18 19:44:04.605418292 +0000 UTC m=+352.207781474"
Sep 18 19:44:04.927724 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:04.927689    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f17f8c7b-46f0-432c-a8cb-281446c61640" path="/var/lib/kubelet/pods/f17f8c7b-46f0-432c-a8cb-281446c61640/volumes"
Sep 18 19:44:18.539633 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:18.539590    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:35.597748 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:35.597717    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-b9b5d54f8-cgnvb"
Sep 18 19:44:36.775756 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:36.775712    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:36.853506 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:36.853465    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.202750 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:37.202717    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.303560 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:37.303523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.404497 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:37.404438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.707236 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:37.707206    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.764475 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:37.764440    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:40.946474 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.946435    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74969fdc99-5w84g"]
Sep 18 19:44:40.946974 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.946909    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f17f8c7b-46f0-432c-a8cb-281446c61640" containerName="manager"
Sep 18 19:44:40.946974 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.946927    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f17f8c7b-46f0-432c-a8cb-281446c61640" containerName="manager"
Sep 18 19:44:40.947089 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.947006    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f17f8c7b-46f0-432c-a8cb-281446c61640" containerName="manager"
Sep 18 19:44:40.950171 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.950120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:40.961391 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.961361    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74969fdc99-5w84g"]
Sep 18 19:44:40.970942 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.970905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-service-ca\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:40.971119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.970956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-oauth-config\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:40.971119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.971032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcj4\" (UniqueName: \"kubernetes.io/projected/db887214-22a8-4b8a-b04f-c80a2e02d3aa-kube-api-access-8tcj4\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:40.971119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.971070    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-oauth-serving-cert\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:40.971119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.971093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-serving-cert\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:40.971119 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.971119    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-config\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:40.971400 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:40.971135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-trusted-ca-bundle\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.005397 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.005358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:41.071875 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.071808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-service-ca\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.071897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-oauth-config\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.071935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcj4\" (UniqueName: \"kubernetes.io/projected/db887214-22a8-4b8a-b04f-c80a2e02d3aa-kube-api-access-8tcj4\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.071969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-oauth-serving-cert\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.071991    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-serving-cert\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.072019    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-config\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.072042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-trusted-ca-bundle\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072606 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.072575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-service-ca\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072721 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.072618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-oauth-serving-cert\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.072836 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.072812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-config\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.073006 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.072983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db887214-22a8-4b8a-b04f-c80a2e02d3aa-trusted-ca-bundle\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.074564 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.074537    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-oauth-config\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.074687 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.074586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db887214-22a8-4b8a-b04f-c80a2e02d3aa-console-serving-cert\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.079306 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.079280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcj4\" (UniqueName: \"kubernetes.io/projected/db887214-22a8-4b8a-b04f-c80a2e02d3aa-kube-api-access-8tcj4\") pod \"console-74969fdc99-5w84g\" (UID: \"db887214-22a8-4b8a-b04f-c80a2e02d3aa\") " pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.147378 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.147337    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:41.248215 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.248122    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:41.260976 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.260936    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:41.533815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.533754    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74969fdc99-5w84g"]
Sep 18 19:44:41.536020 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:44:41.535992    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb887214_22a8_4b8a_b04f_c80a2e02d3aa.slice/crio-964fb9c784be4a95ad38164ccab1afc266f9e93d5fcd4fa74cd3683381f02b7a WatchSource:0}: Error finding container 964fb9c784be4a95ad38164ccab1afc266f9e93d5fcd4fa74cd3683381f02b7a: Status 404 returned error can't find the container with id 964fb9c784be4a95ad38164ccab1afc266f9e93d5fcd4fa74cd3683381f02b7a
Sep 18 19:44:41.747183 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.747149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74969fdc99-5w84g" event={"ID":"db887214-22a8-4b8a-b04f-c80a2e02d3aa","Type":"ContainerStarted","Data":"4e5968b82e62e1951fad23f0aee60a8ae09666010e9b2c9a632cd287a4bda78b"}
Sep 18 19:44:41.747385 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.747204    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74969fdc99-5w84g" event={"ID":"db887214-22a8-4b8a-b04f-c80a2e02d3aa","Type":"ContainerStarted","Data":"964fb9c784be4a95ad38164ccab1afc266f9e93d5fcd4fa74cd3683381f02b7a"}
Sep 18 19:44:41.747643 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.747603    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:41.772754 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.772702    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74969fdc99-5w84g" podStartSLOduration=1.772684912 podStartE2EDuration="1.772684912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:41.77265743 +0000 UTC m=+389.375020611" watchObservedRunningTime="2026-09-18 19:44:41.772684912 +0000 UTC m=+389.375048076"
Sep 18 19:44:41.848460 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:41.848367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:51.261166 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:51.261128    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:51.261653 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:51.261201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:51.266020 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:51.265991    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:51.792992 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:51.792955    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74969fdc99-5w84g"
Sep 18 19:44:51.839538 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:51.839504    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6964f4545-j62vd"]
Sep 18 19:44:51.941400 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:51.941359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:52.043722 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:52.043632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:52.143829 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:44:52.143783    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:16.858891 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:16.858812    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6964f4545-j62vd" podUID="e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" containerName="console" containerID="cri-o://77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84" gracePeriod=15
Sep 18 19:45:17.099416 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.099383    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:45:17.285556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.285518    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-service-ca\") pod \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") "
Sep 18 19:45:17.285755 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.285569    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-oauth-config\") pod \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") "
Sep 18 19:45:17.285755 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.285607    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-config\") pod \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") "
Sep 18 19:45:17.285755 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.285652    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-oauth-serving-cert\") pod \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") "
Sep 18 19:45:17.285755 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.285670    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chqfl\" (UniqueName: \"kubernetes.io/projected/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-kube-api-access-chqfl\") pod \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") "
Sep 18 19:45:17.285755 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.285688    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-serving-cert\") pod \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") "
Sep 18 19:45:17.285755 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.285712    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-trusted-ca-bundle\") pod \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\" (UID: \"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310\") "
Sep 18 19:45:17.286165 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.286019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-service-ca" (OuterVolumeSpecName: "service-ca") pod "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" (UID: "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:17.286337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.286299    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-config" (OuterVolumeSpecName: "console-config") pod "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" (UID: "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:17.286337 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.286309    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" (UID: "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:17.286474 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.286378    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" (UID: "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:17.288070 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.288043    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-kube-api-access-chqfl" (OuterVolumeSpecName: "kube-api-access-chqfl") pod "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" (UID: "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310"). InnerVolumeSpecName "kube-api-access-chqfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:17.288327 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.288297    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" (UID: "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:17.288418 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.288390    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" (UID: "e64fa3e4-2b5b-4d1d-99b8-72ac85a11310"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:17.386276 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.386244    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chqfl\" (UniqueName: \"kubernetes.io/projected/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-kube-api-access-chqfl\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.386276 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.386271    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-serving-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.386276 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.386281    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-trusted-ca-bundle\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.386502 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.386290    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-service-ca\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.386502 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.386299    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-oauth-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.386502 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.386307    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-console-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.386502 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.386315    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310-oauth-serving-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.902757 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.902721    2560 generic.go:356] "Generic (PLEG): container finished" podID="e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" containerID="77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84" exitCode=0
Sep 18 19:45:17.903167 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.902788    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6964f4545-j62vd"
Sep 18 19:45:17.903167 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.902809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6964f4545-j62vd" event={"ID":"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310","Type":"ContainerDied","Data":"77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84"}
Sep 18 19:45:17.903167 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.902912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6964f4545-j62vd" event={"ID":"e64fa3e4-2b5b-4d1d-99b8-72ac85a11310","Type":"ContainerDied","Data":"8340d096d92e8290af83d486860e14cf1e99b003495eddd8ddf5f5ef17a4cad3"}
Sep 18 19:45:17.903167 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.902942    2560 scope.go:117] "RemoveContainer" containerID="77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84"
Sep 18 19:45:17.912248 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.912230    2560 scope.go:117] "RemoveContainer" containerID="77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84"
Sep 18 19:45:17.912579 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:45:17.912560    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84\": container with ID starting with 77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84 not found: ID does not exist" containerID="77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84"
Sep 18 19:45:17.912626 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.912589    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84"} err="failed to get container status \"77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84\": rpc error: code = NotFound desc = could not find container \"77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84\": container with ID starting with 77240596f263ae015cf5c03b03c4f661dd070f83798b8d868522b491f5dfff84 not found: ID does not exist"
Sep 18 19:45:17.924571 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.924546    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6964f4545-j62vd"]
Sep 18 19:45:17.927877 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:17.927843    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6964f4545-j62vd"]
Sep 18 19:45:18.926708 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:18.926664    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" path="/var/lib/kubelet/pods/e64fa3e4-2b5b-4d1d-99b8-72ac85a11310/volumes"
Sep 18 19:45:26.420922 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:26.420838    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": dial tcp 10.133.0.26:8081: connect: connection refused"
Sep 18 19:45:26.948060 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:26.948015    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="96bda9a17582683e862ad1f996790a0428bf28b5ecc643264f057674cde85646" exitCode=1
Sep 18 19:45:26.948229 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:26.948127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"96bda9a17582683e862ad1f996790a0428bf28b5ecc643264f057674cde85646"}
Sep 18 19:45:26.948568 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:26.948549    2560 scope.go:117] "RemoveContainer" containerID="96bda9a17582683e862ad1f996790a0428bf28b5ecc643264f057674cde85646"
Sep 18 19:45:27.953778 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:27.953743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"1a5a02a5633dbecf16e4d7b1e618e0183b74b2de8751b27dc16c88199ff4726d"}
Sep 18 19:45:27.954207 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:27.953987    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:45:46.051343 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:46.051254    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:58.961320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:45:58.961286    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:46:13.611085 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:13.611042    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:18.982960 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.982919    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"]
Sep 18 19:46:18.983362 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.983245    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" containerName="console"
Sep 18 19:46:18.983362 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.983256    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" containerName="console"
Sep 18 19:46:18.983362 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.983315    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64fa3e4-2b5b-4d1d-99b8-72ac85a11310" containerName="console"
Sep 18 19:46:18.987279 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.987261    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:18.990676 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.990652    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xnn8t\""
Sep 18 19:46:18.990818 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.990652    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 19:46:18.990818 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.990656    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 18 19:46:18.993312 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:18.993281    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"]
Sep 18 19:46:19.070069 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.070024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5294g\" (UniqueName: \"kubernetes.io/projected/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-kube-api-access-5294g\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.070251 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.070083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-data\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.070251 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.070103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.170539 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.170505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5294g\" (UniqueName: \"kubernetes.io/projected/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-kube-api-access-5294g\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.170739 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.170559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-data\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.170739 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.170587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.171002 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.170976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-data\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.173133 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.173109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.178497 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.178475    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5294g\" (UniqueName: \"kubernetes.io/projected/0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e-kube-api-access-5294g\") pod \"seaweedfs-tls-custom-8b44d767d-w9lpg\" (UID: \"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.297368 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.297289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"
Sep 18 19:46:19.564820 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:19.564728    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-w9lpg"]
Sep 18 19:46:19.569028 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:46:19.568983    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4ca5c3_a9cf_4957_90b6_e29e91f9c80e.slice/crio-fac9efcc604166e8c97fd284df1d7dcec8848af1d7d4f9f70b812def1aca7136 WatchSource:0}: Error finding container fac9efcc604166e8c97fd284df1d7dcec8848af1d7d4f9f70b812def1aca7136: Status 404 returned error can't find the container with id fac9efcc604166e8c97fd284df1d7dcec8848af1d7d4f9f70b812def1aca7136
Sep 18 19:46:20.175153 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:20.175114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg" event={"ID":"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e","Type":"ContainerStarted","Data":"fac9efcc604166e8c97fd284df1d7dcec8848af1d7d4f9f70b812def1aca7136"}
Sep 18 19:46:23.191348 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:23.191304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg" event={"ID":"0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e","Type":"ContainerStarted","Data":"6ffc904013e11dd428448ec4df11f0e81ed96108dd9b52602e2b183409d9d4df"}
Sep 18 19:46:23.219899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:23.219829    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-w9lpg" podStartSLOduration=2.6405006220000002 podStartE2EDuration="5.219813985s" totalImagesPullingTime="2.579313363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:23.218636074 +0000 UTC m=+490.820999257" watchObservedRunningTime="2026-09-18 19:46:23.219813985 +0000 UTC m=+490.822177168"
Sep 18 19:46:35.565719 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.565686    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"]
Sep 18 19:46:35.568824 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.568804    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.571403 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.571370    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 18 19:46:35.571536 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.571405    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 18 19:46:35.576752 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.576722    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"]
Sep 18 19:46:35.577352 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.577324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:35.601175 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.601138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87r9\" (UniqueName: \"kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-kube-api-access-c87r9\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.601362 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.601209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.601362 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.601237    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.702364 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.702321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.702517 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.702370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.702517 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.702419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87r9\" (UniqueName: \"kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-kube-api-access-c87r9\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.702517 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:46:35.702475    2560 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Sep 18 19:46:35.702517 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:46:35.702497    2560 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-rnb82: secret "seaweedfs-tls-serving" not found
Sep 18 19:46:35.702660 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:46:35.702555    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-seaweedfs-tls-serving podName:de7e8ed5-55e5-4a2b-9104-46dfdb71530f nodeName:}" failed. No retries permitted until 2026-09-18 19:46:36.202539222 +0000 UTC m=+503.804902382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-rnb82" (UID: "de7e8ed5-55e5-4a2b-9104-46dfdb71530f") : secret "seaweedfs-tls-serving" not found
Sep 18 19:46:35.702895 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.702879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.712780 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.712743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87r9\" (UniqueName: \"kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-kube-api-access-c87r9\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:35.834765 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:35.834675    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:36.206556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:36.206517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:36.208941 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:36.208908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/de7e8ed5-55e5-4a2b-9104-46dfdb71530f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-rnb82\" (UID: \"de7e8ed5-55e5-4a2b-9104-46dfdb71530f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:36.479583 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:36.479479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"
Sep 18 19:46:36.748071 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:36.747991    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-rnb82"]
Sep 18 19:46:36.751315 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:46:36.751286    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7e8ed5_55e5_4a2b_9104_46dfdb71530f.slice/crio-33cfcdced8aac27a6695a9bc06a37c28e4ebefa2823b169819bf61308392cc10 WatchSource:0}: Error finding container 33cfcdced8aac27a6695a9bc06a37c28e4ebefa2823b169819bf61308392cc10: Status 404 returned error can't find the container with id 33cfcdced8aac27a6695a9bc06a37c28e4ebefa2823b169819bf61308392cc10
Sep 18 19:46:37.252186 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:37.252074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82" event={"ID":"de7e8ed5-55e5-4a2b-9104-46dfdb71530f","Type":"ContainerStarted","Data":"2483caebd1b1f221307c673984d8c8413347c60f36b0b495d6860e9980b50ceb"}
Sep 18 19:46:37.252186 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:37.252136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82" event={"ID":"de7e8ed5-55e5-4a2b-9104-46dfdb71530f","Type":"ContainerStarted","Data":"33cfcdced8aac27a6695a9bc06a37c28e4ebefa2823b169819bf61308392cc10"}
Sep 18 19:46:37.266552 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:37.266497    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rnb82" podStartSLOduration=2.02163646 podStartE2EDuration="2.266479814s" totalImagesPullingTime="244.843354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:37.265796693 +0000 UTC m=+504.868159876" watchObservedRunningTime="2026-09-18 19:46:37.266479814 +0000 UTC m=+504.868842996"
Sep 18 19:46:49.438369 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:49.438290    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:49.539098 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:46:49.539052    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.433586 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:00.433550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.617373 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:00.617332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.702533 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:00.702442    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:01.422385 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:01.422354    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:15.384547 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:15.384449    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:20.575318 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:20.570943    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:20.804705 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:20.804674    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:21.762784 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:21.762743    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:44.550330 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:44.550294    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="1a5a02a5633dbecf16e4d7b1e618e0183b74b2de8751b27dc16c88199ff4726d" exitCode=1
Sep 18 19:47:44.550742 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:44.550370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"1a5a02a5633dbecf16e4d7b1e618e0183b74b2de8751b27dc16c88199ff4726d"}
Sep 18 19:47:44.550742 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:44.550454    2560 scope.go:117] "RemoveContainer" containerID="96bda9a17582683e862ad1f996790a0428bf28b5ecc643264f057674cde85646"
Sep 18 19:47:44.550900 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:44.550782    2560 scope.go:117] "RemoveContainer" containerID="1a5a02a5633dbecf16e4d7b1e618e0183b74b2de8751b27dc16c88199ff4726d"
Sep 18 19:47:44.551036 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:47:44.551012    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:47:48.958897 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:48.958824    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:47:48.959428 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:48.959324    2560 scope.go:117] "RemoveContainer" containerID="1a5a02a5633dbecf16e4d7b1e618e0183b74b2de8751b27dc16c88199ff4726d"
Sep 18 19:47:48.959583 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:47:48.959558    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:47:52.047123 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:52.047077    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:47:52.047607 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:47:52.047462    2560 scope.go:117] "RemoveContainer" containerID="1a5a02a5633dbecf16e4d7b1e618e0183b74b2de8751b27dc16c88199ff4726d"
Sep 18 19:47:52.047652 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:47:52.047634    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:48:05.922572 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:48:05.922535    2560 scope.go:117] "RemoveContainer" containerID="1a5a02a5633dbecf16e4d7b1e618e0183b74b2de8751b27dc16c88199ff4726d"
Sep 18 19:48:08.662496 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:48:08.662461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"21bd4a9bb5eddd24366c213e9ca0484ad20d89e0146a7486e80714648fcd497f"}
Sep 18 19:48:08.662924 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:48:08.662804    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:48:39.669228 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:48:39.669144    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:48:55.554948 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:48:55.554512    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:56.676890 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:48:56.676834    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:05.686655 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:49:05.686623    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:21.065392 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:49:21.065340    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:26.200661 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:49:26.200620    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:26.540266 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:49:26.540172    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:27.267449 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:27.267414    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="21bd4a9bb5eddd24366c213e9ca0484ad20d89e0146a7486e80714648fcd497f" exitCode=1
Sep 18 19:50:27.267989 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:27.267483    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"21bd4a9bb5eddd24366c213e9ca0484ad20d89e0146a7486e80714648fcd497f"}
Sep 18 19:50:27.267989 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:27.267545    2560 scope.go:117] "RemoveContainer" containerID="1a5a02a5633dbecf16e4d7b1e618e0183b74b2de8751b27dc16c88199ff4726d"
Sep 18 19:50:27.267989 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:27.267932    2560 scope.go:117] "RemoveContainer" containerID="21bd4a9bb5eddd24366c213e9ca0484ad20d89e0146a7486e80714648fcd497f"
Sep 18 19:50:27.268189 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:50:27.268168    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:50:29.667003 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:29.666953    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:50:29.667415 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:29.667307    2560 scope.go:117] "RemoveContainer" containerID="21bd4a9bb5eddd24366c213e9ca0484ad20d89e0146a7486e80714648fcd497f"
Sep 18 19:50:29.667495 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:50:29.667476    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:50:30.632976 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:30.632935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:30.707735 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:30.707692    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:31.174406 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:31.174369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:32.046114 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:32.046071    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:50:32.046628 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:32.046596    2560 scope.go:117] "RemoveContainer" containerID="21bd4a9bb5eddd24366c213e9ca0484ad20d89e0146a7486e80714648fcd497f"
Sep 18 19:50:32.046823 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:50:32.046798    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:50:46.922661 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:46.922622    2560 scope.go:117] "RemoveContainer" containerID="21bd4a9bb5eddd24366c213e9ca0484ad20d89e0146a7486e80714648fcd497f"
Sep 18 19:50:46.923305 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:46.923068    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:50:48.362419 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:48.362384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"}
Sep 18 19:50:48.362807 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:48.362691    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:50:50.717220 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:50.717182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:51.273202 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:51.273169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:56.856061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:50:56.856020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:02.002285 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:51:02.002208    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:02.127697 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:51:02.127657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:19.368787 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:51:19.368753    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:51:46.348793 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:51:46.348681    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:25.931582 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:25.931528    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:26.030058 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:26.030017    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:26.093399 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:26.093358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:26.452769 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:26.452728    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:29.628387 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:29.628358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:42.854085 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:42.854044    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:45.705198 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:45.705154    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:45.757066 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:45.757035    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:47.988103 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:47.988068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:48.012493 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.012456    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"]
Sep 18 19:52:48.015997 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.015971    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.018380 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.018357    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 19:52:48.018510 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.018399    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 18 19:52:48.019059 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.019040    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:52:48.019161 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.019041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:52:48.019161 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.019108    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:52:48.019422 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.019397    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:52:48.019536 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.019454    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:52:48.028050 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.028024    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"]
Sep 18 19:52:48.065787 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.065750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.065974 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.065799    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.065974 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.065867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd64t\" (UniqueName: \"kubernetes.io/projected/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-api-access-gd64t\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.065974 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.065890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.065974 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.065917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/775c7e33-2905-4302-a1ea-6d3521a55b94-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.166320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.166289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/775c7e33-2905-4302-a1ea-6d3521a55b94-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.166517 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.166334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.166517 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.166365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.166517 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.166405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd64t\" (UniqueName: \"kubernetes.io/projected/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-api-access-gd64t\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.166517 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.166423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.166743 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:52:48.166522    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 18 19:52:48.166743 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:52:48.166635    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls podName:775c7e33-2905-4302-a1ea-6d3521a55b94 nodeName:}" failed. No retries permitted until 2026-09-18 19:52:48.666615163 +0000 UTC m=+876.268978332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" (UID: "775c7e33-2905-4302-a1ea-6d3521a55b94") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 18 19:52:48.166879 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.166784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/775c7e33-2905-4302-a1ea-6d3521a55b94-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.167207 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.167185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.167247 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.167185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.175357 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.175326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd64t\" (UniqueName: \"kubernetes.io/projected/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-api-access-gd64t\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.536635 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.536592    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:48.669253 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.669217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.671783 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.671754    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:48.929031 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:48.929003    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:52:49.204360 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:49.204276    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"]
Sep 18 19:52:49.207267 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:52:49.207238    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775c7e33_2905_4302_a1ea_6d3521a55b94.slice/crio-5732d67ff7bd06260dc75368b7279010554c42f6e873898921f3dcb19d069e29 WatchSource:0}: Error finding container 5732d67ff7bd06260dc75368b7279010554c42f6e873898921f3dcb19d069e29: Status 404 returned error can't find the container with id 5732d67ff7bd06260dc75368b7279010554c42f6e873898921f3dcb19d069e29
Sep 18 19:52:49.893500 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:49.893465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" event={"ID":"775c7e33-2905-4302-a1ea-6d3521a55b94","Type":"ContainerStarted","Data":"5732d67ff7bd06260dc75368b7279010554c42f6e873898921f3dcb19d069e29"}
Sep 18 19:52:53.914113 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:53.914074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" event={"ID":"775c7e33-2905-4302-a1ea-6d3521a55b94","Type":"ContainerStarted","Data":"69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0"}
Sep 18 19:52:56.930046 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:56.930008    2560 generic.go:356] "Generic (PLEG): container finished" podID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerID="69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0" exitCode=0
Sep 18 19:52:56.930511 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:52:56.930051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" event={"ID":"775c7e33-2905-4302-a1ea-6d3521a55b94","Type":"ContainerDied","Data":"69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0"}
Sep 18 19:53:08.007825 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:08.007789    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e" exitCode=1
Sep 18 19:53:08.008313 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:08.007889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"}
Sep 18 19:53:08.008313 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:08.007955    2560 scope.go:117] "RemoveContainer" containerID="21bd4a9bb5eddd24366c213e9ca0484ad20d89e0146a7486e80714648fcd497f"
Sep 18 19:53:08.008436 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:08.008366    2560 scope.go:117] "RemoveContainer" containerID="54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"
Sep 18 19:53:08.008877 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:53:08.008586    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:53:09.367376 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:09.367338    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:53:09.367817 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:09.367700    2560 scope.go:117] "RemoveContainer" containerID="54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"
Sep 18 19:53:09.367938 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:53:09.367905    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:53:12.046888 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:12.046832    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:53:12.047330 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:12.047318    2560 scope.go:117] "RemoveContainer" containerID="54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"
Sep 18 19:53:12.047534 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:53:12.047512    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:53:20.072787 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:20.072742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" event={"ID":"775c7e33-2905-4302-a1ea-6d3521a55b94","Type":"ContainerStarted","Data":"0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e"}
Sep 18 19:53:22.085834 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:22.085744    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" event={"ID":"775c7e33-2905-4302-a1ea-6d3521a55b94","Type":"ContainerStarted","Data":"cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1"}
Sep 18 19:53:22.086318 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:22.086070    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:53:22.110927 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:22.110841    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podStartSLOduration=2.938273425 podStartE2EDuration="35.110824101s" totalImagesPullingTime="28.515576747s" totalInitContainerRuntime="3.656973929s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:22.108927767 +0000 UTC m=+909.711290950" watchObservedRunningTime="2026-09-18 19:53:22.110824101 +0000 UTC m=+909.713187282"
Sep 18 19:53:22.924413 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:22.924381    2560 scope.go:117] "RemoveContainer" containerID="54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"
Sep 18 19:53:22.924605 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:53:22.924581    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:53:23.090248 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:23.090218    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:53:29.101639 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:29.101609    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:53:36.922687 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:36.922653    2560 scope.go:117] "RemoveContainer" containerID="54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"
Sep 18 19:53:36.923253 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:53:36.922811    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:53:48.922926 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:48.922889    2560 scope.go:117] "RemoveContainer" containerID="54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"
Sep 18 19:53:49.103126 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:49.103086    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:53:50.218570 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:50.218527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"}
Sep 18 19:53:50.219057 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:50.218779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:53:59.102834 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:53:59.102783    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:09.103029 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:09.102989    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:19.103062 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:19.103015    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:21.225798 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:21.225764    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:54:29.102748 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:29.102704    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:39.681560 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:39.681474    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:54:48.143651 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:48.143617    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"]
Sep 18 19:54:48.144201 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:48.143889    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container" containerID="cri-o://0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e" gracePeriod=30
Sep 18 19:54:48.144201 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:48.143924    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kube-rbac-proxy" containerID="cri-o://cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1" gracePeriod=30
Sep 18 19:54:48.186865 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:48.186828    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:48.475139 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:48.475102    2560 generic.go:356] "Generic (PLEG): container finished" podID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerID="cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1" exitCode=2
Sep 18 19:54:48.475310 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:48.475177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" event={"ID":"775c7e33-2905-4302-a1ea-6d3521a55b94","Type":"ContainerDied","Data":"cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1"}
Sep 18 19:54:49.095800 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:49.095755    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 19:54:49.102173 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:49.102133    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:52.338517 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:52.338475    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:53.244015 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.243971    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:53.288351 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.288314    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"]
Sep 18 19:54:53.292802 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.292777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.295067 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.295044    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 18 19:54:53.295170 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.295086    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:54:53.301281 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.301251    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"]
Sep 18 19:54:53.401992 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.401953    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:53.417371 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.417334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrk7\" (UniqueName: \"kubernetes.io/projected/7cad6a91-069d-455f-8575-784f23b6a971-kube-api-access-rkrk7\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.417758 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.417382    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.417758 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.417463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cad6a91-069d-455f-8575-784f23b6a971-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.417758 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.417507    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cad6a91-069d-455f-8575-784f23b6a971-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.417758 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.417563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.497002 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.496913    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:54:53.498697 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.498671    2560 generic.go:356] "Generic (PLEG): container finished" podID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerID="0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e" exitCode=0
Sep 18 19:54:53.498815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.498747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" event={"ID":"775c7e33-2905-4302-a1ea-6d3521a55b94","Type":"ContainerDied","Data":"0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e"}
Sep 18 19:54:53.498882 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.498826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz" event={"ID":"775c7e33-2905-4302-a1ea-6d3521a55b94","Type":"ContainerDied","Data":"5732d67ff7bd06260dc75368b7279010554c42f6e873898921f3dcb19d069e29"}
Sep 18 19:54:53.498882 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.498876    2560 scope.go:117] "RemoveContainer" containerID="cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1"
Sep 18 19:54:53.506936 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.506915    2560 scope.go:117] "RemoveContainer" containerID="0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e"
Sep 18 19:54:53.516451 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.516431    2560 scope.go:117] "RemoveContainer" containerID="69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0"
Sep 18 19:54:53.518817 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.518793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrk7\" (UniqueName: \"kubernetes.io/projected/7cad6a91-069d-455f-8575-784f23b6a971-kube-api-access-rkrk7\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.518933 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.518827    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.518933 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.518904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cad6a91-069d-455f-8575-784f23b6a971-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.519102 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.518943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cad6a91-069d-455f-8575-784f23b6a971-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.519102 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.518987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.519473 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.519449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cad6a91-069d-455f-8575-784f23b6a971-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.519637 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.519616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.519695 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.519679    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.521481 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.521463    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cad6a91-069d-455f-8575-784f23b6a971-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.526421 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.526397    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrk7\" (UniqueName: \"kubernetes.io/projected/7cad6a91-069d-455f-8575-784f23b6a971-kube-api-access-rkrk7\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.537533 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.537510    2560 scope.go:117] "RemoveContainer" containerID="cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1"
Sep 18 19:54:53.537923 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:54:53.537896    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1\": container with ID starting with cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1 not found: ID does not exist" containerID="cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1"
Sep 18 19:54:53.538005 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.537931    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1"} err="failed to get container status \"cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1\": rpc error: code = NotFound desc = could not find container \"cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1\": container with ID starting with cace0b7514184133a2468d6dfde8c2438eb2c26522cfa1b72cf2a7f302529dc1 not found: ID does not exist"
Sep 18 19:54:53.538005 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.537952    2560 scope.go:117] "RemoveContainer" containerID="0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e"
Sep 18 19:54:53.538272 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:54:53.538248    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e\": container with ID starting with 0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e not found: ID does not exist" containerID="0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e"
Sep 18 19:54:53.538339 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.538282    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e"} err="failed to get container status \"0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e\": rpc error: code = NotFound desc = could not find container \"0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e\": container with ID starting with 0f4b318cfe5ce9d513ef8f7069925e8dbdd14f22d91c4f99b3b0da3102f48f1e not found: ID does not exist"
Sep 18 19:54:53.538339 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.538307    2560 scope.go:117] "RemoveContainer" containerID="69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0"
Sep 18 19:54:53.538591 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:54:53.538574    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0\": container with ID starting with 69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0 not found: ID does not exist" containerID="69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0"
Sep 18 19:54:53.538639 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.538598    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0"} err="failed to get container status \"69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0\": rpc error: code = NotFound desc = could not find container \"69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0\": container with ID starting with 69d05587d0e7475d6f0b863de132c4925c96efba94f8323176780d352cd533b0 not found: ID does not exist"
Sep 18 19:54:53.619562 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.619518    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-rbac-proxy-sar-config\") pod \"775c7e33-2905-4302-a1ea-6d3521a55b94\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") "
Sep 18 19:54:53.619763 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.619571    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls\") pod \"775c7e33-2905-4302-a1ea-6d3521a55b94\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") "
Sep 18 19:54:53.619763 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.619629    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/775c7e33-2905-4302-a1ea-6d3521a55b94-kserve-provision-location\") pod \"775c7e33-2905-4302-a1ea-6d3521a55b94\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") "
Sep 18 19:54:53.619763 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.619667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-cabundle-cert\") pod \"775c7e33-2905-4302-a1ea-6d3521a55b94\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") "
Sep 18 19:54:53.619763 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.619712    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd64t\" (UniqueName: \"kubernetes.io/projected/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-api-access-gd64t\") pod \"775c7e33-2905-4302-a1ea-6d3521a55b94\" (UID: \"775c7e33-2905-4302-a1ea-6d3521a55b94\") "
Sep 18 19:54:53.620045 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.620015    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "775c7e33-2905-4302-a1ea-6d3521a55b94" (UID: "775c7e33-2905-4302-a1ea-6d3521a55b94"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:53.620045 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.620037    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c7e33-2905-4302-a1ea-6d3521a55b94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "775c7e33-2905-4302-a1ea-6d3521a55b94" (UID: "775c7e33-2905-4302-a1ea-6d3521a55b94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:53.620172 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.620060    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "775c7e33-2905-4302-a1ea-6d3521a55b94" (UID: "775c7e33-2905-4302-a1ea-6d3521a55b94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:53.621920 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.621890    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "775c7e33-2905-4302-a1ea-6d3521a55b94" (UID: "775c7e33-2905-4302-a1ea-6d3521a55b94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:54:53.621998 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.621987    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-api-access-gd64t" (OuterVolumeSpecName: "kube-api-access-gd64t") pod "775c7e33-2905-4302-a1ea-6d3521a55b94" (UID: "775c7e33-2905-4302-a1ea-6d3521a55b94"). InnerVolumeSpecName "kube-api-access-gd64t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:53.674198 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.674154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:53.721285 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.721243    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/775c7e33-2905-4302-a1ea-6d3521a55b94-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:54:53.721285 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.721279    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/775c7e33-2905-4302-a1ea-6d3521a55b94-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:54:53.721504 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.721293    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:54:53.721504 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.721305    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd64t\" (UniqueName: \"kubernetes.io/projected/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-api-access-gd64t\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:54:53.721504 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.721313    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/775c7e33-2905-4302-a1ea-6d3521a55b94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:54:53.946744 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:53.946717    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"]
Sep 18 19:54:53.948634 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:54:53.948605    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cad6a91_069d_455f_8575_784f23b6a971.slice/crio-5f6651bde90bd11f61b6ae98876155cc930143b71d6631fb90a3ed8b15e02b78 WatchSource:0}: Error finding container 5f6651bde90bd11f61b6ae98876155cc930143b71d6631fb90a3ed8b15e02b78: Status 404 returned error can't find the container with id 5f6651bde90bd11f61b6ae98876155cc930143b71d6631fb90a3ed8b15e02b78
Sep 18 19:54:54.505108 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:54.505077    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"
Sep 18 19:54:54.507038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:54.507002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" event={"ID":"7cad6a91-069d-455f-8575-784f23b6a971","Type":"ContainerStarted","Data":"5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b"}
Sep 18 19:54:54.507188 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:54.507065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" event={"ID":"7cad6a91-069d-455f-8575-784f23b6a971","Type":"ContainerStarted","Data":"5f6651bde90bd11f61b6ae98876155cc930143b71d6631fb90a3ed8b15e02b78"}
Sep 18 19:54:54.543877 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:54.543822    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"]
Sep 18 19:54:54.548275 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:54.548195    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz"]
Sep 18 19:54:54.927976 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:54.927933    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" path="/var/lib/kubelet/pods/775c7e33-2905-4302-a1ea-6d3521a55b94/volumes"
Sep 18 19:54:57.457966 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:57.457935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:57.717203 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:57.717100    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:58.527008 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:58.526976    2560 generic.go:356] "Generic (PLEG): container finished" podID="7cad6a91-069d-455f-8575-784f23b6a971" containerID="5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b" exitCode=0
Sep 18 19:54:58.527405 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:58.527023    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" event={"ID":"7cad6a91-069d-455f-8575-784f23b6a971","Type":"ContainerDied","Data":"5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b"}
Sep 18 19:54:59.533543 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:59.533504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" event={"ID":"7cad6a91-069d-455f-8575-784f23b6a971","Type":"ContainerStarted","Data":"550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17"}
Sep 18 19:54:59.533979 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:59.533572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" event={"ID":"7cad6a91-069d-455f-8575-784f23b6a971","Type":"ContainerStarted","Data":"d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7"}
Sep 18 19:54:59.533979 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:59.533784    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:54:59.551912 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:54:59.551840    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podStartSLOduration=2.826770979 podStartE2EDuration="6.551819863s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725048884s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:59.550896189 +0000 UTC m=+1007.153259373" watchObservedRunningTime="2026-09-18 19:54:59.551819863 +0000 UTC m=+1007.154183042"
Sep 18 19:55:00.537790 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:55:00.537745    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:55:06.547981 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:55:06.547945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:55:34.425397 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:55:34.425357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:36.549164 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:55:36.549118    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:55:46.549223 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:55:46.549177    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:55:56.548724 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:55:56.548680    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:06.549320 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:06.549221    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:06.827750 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:06.827660    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603" exitCode=1
Sep 18 19:56:06.827750 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:06.827718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"}
Sep 18 19:56:06.827979 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:06.827783    2560 scope.go:117] "RemoveContainer" containerID="54b74e21b529a01cd454c75f2ac18fb22234aa7bfc6488d46b9d43deea82f98e"
Sep 18 19:56:06.828122 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:06.828101    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:56:06.828336 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:06.828316    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:56:11.224797 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:11.224752    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:56:11.225333 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:11.225255    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:56:11.225512 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:11.225488    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:56:12.046459 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:12.046407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:56:12.046815 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:12.046801    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:56:12.047053 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:12.047032    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:56:16.273442 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:16.273400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:17.178963 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:17.178928    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:56:22.924094 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:22.924062    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:56:22.924469 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:22.924312    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:56:23.364622 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:23.364540    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"]
Sep 18 19:56:23.364802 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:23.364775    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container" containerID="cri-o://d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7" gracePeriod=30
Sep 18 19:56:23.364882 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:23.364826    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kube-rbac-proxy" containerID="cri-o://550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17" gracePeriod=30
Sep 18 19:56:23.379350 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:23.379325    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:23.905002 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:23.904957    2560 generic.go:356] "Generic (PLEG): container finished" podID="7cad6a91-069d-455f-8575-784f23b6a971" containerID="550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17" exitCode=2
Sep 18 19:56:23.905178 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:23.905033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" event={"ID":"7cad6a91-069d-455f-8575-784f23b6a971","Type":"ContainerDied","Data":"550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17"}
Sep 18 19:56:26.543419 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:26.543366    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 19:56:27.179574 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:27.179527    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:28.526552 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.526512    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:28.572667 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.572618    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"]
Sep 18 19:56:28.573171 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.573149    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container"
Sep 18 19:56:28.573171 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.573172    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container"
Sep 18 19:56:28.573367 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.573196    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kube-rbac-proxy"
Sep 18 19:56:28.573367 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.573204    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kube-rbac-proxy"
Sep 18 19:56:28.573367 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.573221    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="storage-initializer"
Sep 18 19:56:28.573367 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.573230    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="storage-initializer"
Sep 18 19:56:28.573367 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.573302    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kube-rbac-proxy"
Sep 18 19:56:28.573367 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.573312    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="775c7e33-2905-4302-a1ea-6d3521a55b94" containerName="kserve-container"
Sep 18 19:56:28.576840 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.576816    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.579244 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.579216    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 18 19:56:28.579364 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.579259    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:56:28.585508 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.585477    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"]
Sep 18 19:56:28.694565 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.694530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.694727 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.694576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.694727 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.694620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcptn\" (UniqueName: \"kubernetes.io/projected/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-api-access-qcptn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.694727 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.694642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.694727 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.694665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.796029 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.796001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.796127 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.796092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.796164 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.796140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.796221 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.796204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcptn\" (UniqueName: \"kubernetes.io/projected/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-api-access-qcptn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.796268 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.796236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.796481 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.796450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.796942 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.796922    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.797146 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.797124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.798392 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.798367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:28.798591 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.798575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.803584 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.803558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcptn\" (UniqueName: \"kubernetes.io/projected/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-api-access-qcptn\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.822556 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.822528    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:56:28.897116 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.897075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkrk7\" (UniqueName: \"kubernetes.io/projected/7cad6a91-069d-455f-8575-784f23b6a971-kube-api-access-rkrk7\") pod \"7cad6a91-069d-455f-8575-784f23b6a971\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") "
Sep 18 19:56:28.897318 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.897129    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-cabundle-cert\") pod \"7cad6a91-069d-455f-8575-784f23b6a971\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") "
Sep 18 19:56:28.897318 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.897161    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cad6a91-069d-455f-8575-784f23b6a971-proxy-tls\") pod \"7cad6a91-069d-455f-8575-784f23b6a971\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") "
Sep 18 19:56:28.897318 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.897196    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cad6a91-069d-455f-8575-784f23b6a971-kserve-provision-location\") pod \"7cad6a91-069d-455f-8575-784f23b6a971\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") "
Sep 18 19:56:28.897318 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.897289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-kube-rbac-proxy-sar-config\") pod \"7cad6a91-069d-455f-8575-784f23b6a971\" (UID: \"7cad6a91-069d-455f-8575-784f23b6a971\") "
Sep 18 19:56:28.897588 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.897548    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cad6a91-069d-455f-8575-784f23b6a971-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cad6a91-069d-455f-8575-784f23b6a971" (UID: "7cad6a91-069d-455f-8575-784f23b6a971"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:28.897674 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.897619    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cad6a91-069d-455f-8575-784f23b6a971" (UID: "7cad6a91-069d-455f-8575-784f23b6a971"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:28.897717 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.897684    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cad6a91-069d-455f-8575-784f23b6a971" (UID: "7cad6a91-069d-455f-8575-784f23b6a971"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:28.899463 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.899435    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cad6a91-069d-455f-8575-784f23b6a971-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cad6a91-069d-455f-8575-784f23b6a971" (UID: "7cad6a91-069d-455f-8575-784f23b6a971"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:28.899561 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.899469    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cad6a91-069d-455f-8575-784f23b6a971-kube-api-access-rkrk7" (OuterVolumeSpecName: "kube-api-access-rkrk7") pod "7cad6a91-069d-455f-8575-784f23b6a971" (UID: "7cad6a91-069d-455f-8575-784f23b6a971"). InnerVolumeSpecName "kube-api-access-rkrk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:28.929938 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.929894    2560 generic.go:356] "Generic (PLEG): container finished" podID="7cad6a91-069d-455f-8575-784f23b6a971" containerID="d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7" exitCode=0
Sep 18 19:56:28.930113 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.929956    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"
Sep 18 19:56:28.930113 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.929991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" event={"ID":"7cad6a91-069d-455f-8575-784f23b6a971","Type":"ContainerDied","Data":"d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7"}
Sep 18 19:56:28.930113 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.930062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx" event={"ID":"7cad6a91-069d-455f-8575-784f23b6a971","Type":"ContainerDied","Data":"5f6651bde90bd11f61b6ae98876155cc930143b71d6631fb90a3ed8b15e02b78"}
Sep 18 19:56:28.930113 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.930102    2560 scope.go:117] "RemoveContainer" containerID="550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17"
Sep 18 19:56:28.939272 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.939245    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:28.939693 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.939667    2560 scope.go:117] "RemoveContainer" containerID="d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7"
Sep 18 19:56:28.949281 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.949250    2560 scope.go:117] "RemoveContainer" containerID="5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b"
Sep 18 19:56:28.956193 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.956016    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"]
Sep 18 19:56:28.958652 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.958625    2560 scope.go:117] "RemoveContainer" containerID="550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17"
Sep 18 19:56:28.959257 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:28.959227    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17\": container with ID starting with 550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17 not found: ID does not exist" containerID="550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17"
Sep 18 19:56:28.959374 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.959273    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17"} err="failed to get container status \"550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17\": rpc error: code = NotFound desc = could not find container \"550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17\": container with ID starting with 550408b8cfef7ea057f2f8cdc1b16966035d0c714c5cf2c50134ea6b4c976b17 not found: ID does not exist"
Sep 18 19:56:28.959374 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.959304    2560 scope.go:117] "RemoveContainer" containerID="d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7"
Sep 18 19:56:28.959711 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:28.959683    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7\": container with ID starting with d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7 not found: ID does not exist" containerID="d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7"
Sep 18 19:56:28.959780 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.959721    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7"} err="failed to get container status \"d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7\": rpc error: code = NotFound desc = could not find container \"d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7\": container with ID starting with d9920aee7a01bd04a5dbefcf597dc32b7ae6a9d29995ac9a4d246df2a05d3ca7 not found: ID does not exist"
Sep 18 19:56:28.959780 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.959747    2560 scope.go:117] "RemoveContainer" containerID="5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b"
Sep 18 19:56:28.960113 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:28.960091    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b\": container with ID starting with 5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b not found: ID does not exist" containerID="5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b"
Sep 18 19:56:28.960194 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.960119    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b"} err="failed to get container status \"5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b\": rpc error: code = NotFound desc = could not find container \"5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b\": container with ID starting with 5402308045a918082cb996cf18711767d86bb9e39454e71c35b557b428da101b not found: ID does not exist"
Sep 18 19:56:28.960366 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.960343    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx"]
Sep 18 19:56:28.999326 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.999297    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrk7\" (UniqueName: \"kubernetes.io/projected/7cad6a91-069d-455f-8575-784f23b6a971-kube-api-access-rkrk7\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:56:28.999452 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.999331    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:56:28.999452 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.999348    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cad6a91-069d-455f-8575-784f23b6a971-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:56:28.999452 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.999361    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cad6a91-069d-455f-8575-784f23b6a971-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:56:28.999452 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:28.999373    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cad6a91-069d-455f-8575-784f23b6a971-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:56:29.221223 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:29.221196    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"]
Sep 18 19:56:29.223567 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:56:29.223528    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f5d722_d3bf_4c6d_b25a_34f85e1b59da.slice/crio-23b6879537afbcfa6450e8a4e5ff66766c5b973784564653cda0212d3f1fca78 WatchSource:0}: Error finding container 23b6879537afbcfa6450e8a4e5ff66766c5b973784564653cda0212d3f1fca78: Status 404 returned error can't find the container with id 23b6879537afbcfa6450e8a4e5ff66766c5b973784564653cda0212d3f1fca78
Sep 18 19:56:29.937443 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:29.937406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" event={"ID":"76f5d722-d3bf-4c6d-b25a-34f85e1b59da","Type":"ContainerStarted","Data":"85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52"}
Sep 18 19:56:29.937859 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:29.937464    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" event={"ID":"76f5d722-d3bf-4c6d-b25a-34f85e1b59da","Type":"ContainerStarted","Data":"23b6879537afbcfa6450e8a4e5ff66766c5b973784564653cda0212d3f1fca78"}
Sep 18 19:56:30.927170 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:30.927133    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cad6a91-069d-455f-8575-784f23b6a971" path="/var/lib/kubelet/pods/7cad6a91-069d-455f-8575-784f23b6a971/volumes"
Sep 18 19:56:33.922327 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:33.922291    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:56:33.922676 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:33.922483    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:56:33.957239 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:33.957205    2560 generic.go:356] "Generic (PLEG): container finished" podID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerID="85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52" exitCode=0
Sep 18 19:56:33.957423 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:33.957247    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" event={"ID":"76f5d722-d3bf-4c6d-b25a-34f85e1b59da","Type":"ContainerDied","Data":"85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52"}
Sep 18 19:56:34.964493 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:34.964459    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" event={"ID":"76f5d722-d3bf-4c6d-b25a-34f85e1b59da","Type":"ContainerStarted","Data":"b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f"}
Sep 18 19:56:34.964939 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:34.964520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" event={"ID":"76f5d722-d3bf-4c6d-b25a-34f85e1b59da","Type":"ContainerStarted","Data":"71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b"}
Sep 18 19:56:34.964939 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:34.964746    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:34.964939 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:34.964913    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:34.983085 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:34.983024    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podStartSLOduration=2.9629452389999997 podStartE2EDuration="6.983004329s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.02005909s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:34.981300971 +0000 UTC m=+1102.583664154" watchObservedRunningTime="2026-09-18 19:56:34.983004329 +0000 UTC m=+1102.585367510"
Sep 18 19:56:40.974292 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:40.974257    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:56:45.922267 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:45.922232    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:56:45.922645 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:45.922384    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:56:59.922162 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:56:59.922125    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:56:59.922541 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:56:59.922319    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:57:03.777967 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:03.777926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:08.910410 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:08.910371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:08.980987 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:08.980940    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:10.975728 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:10.975688    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 19:57:11.922620 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:11.922587    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:57:11.922795 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:57:11.922778    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:57:20.975636 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:20.975592    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 19:57:26.923024 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:26.922992    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:57:26.923399 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:26.923272    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:57:28.207474 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:28.207437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"}
Sep 18 19:57:28.207892 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:28.207680    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:57:30.975702 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:30.975657    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 19:57:31.952297 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:31.952260    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:40.974861 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:40.974766    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 19:57:50.975577 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:50.975536    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 19:57:51.922601 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:51.922556    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 19:57:59.214963 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:57:59.214925    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:58:02.483806 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:02.483766    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:58:08.641567 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:08.641533    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"]
Sep 18 19:58:08.642059 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:08.641813    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container" containerID="cri-o://71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b" gracePeriod=30
Sep 18 19:58:08.642059 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:08.641825    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kube-rbac-proxy" containerID="cri-o://b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f" gracePeriod=30
Sep 18 19:58:08.664421 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:08.664392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:09.395974 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:09.395942    2560 generic.go:356] "Generic (PLEG): container finished" podID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerID="b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f" exitCode=2
Sep 18 19:58:09.396163 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:09.396019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" event={"ID":"76f5d722-d3bf-4c6d-b25a-34f85e1b59da","Type":"ContainerDied","Data":"b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f"}
Sep 18 19:58:10.970899 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:10.970825    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 18 19:58:11.923173 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:11.923128    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 19:58:13.808511 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:13.808473    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:14.188075 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.188044    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:58:14.274061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.274029    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:14.283068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.283029    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kserve-provision-location\") pod \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") "
Sep 18 19:58:14.283068 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.283073    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-rbac-proxy-sar-config\") pod \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") "
Sep 18 19:58:14.283289 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.283106    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcptn\" (UniqueName: \"kubernetes.io/projected/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-api-access-qcptn\") pod \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") "
Sep 18 19:58:14.283289 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.283123    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-cabundle-cert\") pod \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") "
Sep 18 19:58:14.283289 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.283146    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-proxy-tls\") pod \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\" (UID: \"76f5d722-d3bf-4c6d-b25a-34f85e1b59da\") "
Sep 18 19:58:14.283512 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.283485    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76f5d722-d3bf-4c6d-b25a-34f85e1b59da" (UID: "76f5d722-d3bf-4c6d-b25a-34f85e1b59da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:14.283575 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.283506    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76f5d722-d3bf-4c6d-b25a-34f85e1b59da" (UID: "76f5d722-d3bf-4c6d-b25a-34f85e1b59da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:58:14.283575 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.283535    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76f5d722-d3bf-4c6d-b25a-34f85e1b59da" (UID: "76f5d722-d3bf-4c6d-b25a-34f85e1b59da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:58:14.285352 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.285328    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-api-access-qcptn" (OuterVolumeSpecName: "kube-api-access-qcptn") pod "76f5d722-d3bf-4c6d-b25a-34f85e1b59da" (UID: "76f5d722-d3bf-4c6d-b25a-34f85e1b59da"). InnerVolumeSpecName "kube-api-access-qcptn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:14.285465 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.285356    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76f5d722-d3bf-4c6d-b25a-34f85e1b59da" (UID: "76f5d722-d3bf-4c6d-b25a-34f85e1b59da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:58:14.384061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.383963    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcptn\" (UniqueName: \"kubernetes.io/projected/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-api-access-qcptn\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:58:14.384061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.383994    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:58:14.384061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.384003    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:58:14.384061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.384011    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:58:14.384061 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.384020    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76f5d722-d3bf-4c6d-b25a-34f85e1b59da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 19:58:14.425840 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.425805    2560 generic.go:356] "Generic (PLEG): container finished" podID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerID="71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b" exitCode=0
Sep 18 19:58:14.426038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.425882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" event={"ID":"76f5d722-d3bf-4c6d-b25a-34f85e1b59da","Type":"ContainerDied","Data":"71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b"}
Sep 18 19:58:14.426038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.425943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp" event={"ID":"76f5d722-d3bf-4c6d-b25a-34f85e1b59da","Type":"ContainerDied","Data":"23b6879537afbcfa6450e8a4e5ff66766c5b973784564653cda0212d3f1fca78"}
Sep 18 19:58:14.426038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.425889    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"
Sep 18 19:58:14.426038 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.425971    2560 scope.go:117] "RemoveContainer" containerID="b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f"
Sep 18 19:58:14.434581 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.434564    2560 scope.go:117] "RemoveContainer" containerID="71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b"
Sep 18 19:58:14.442499 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.442478    2560 scope.go:117] "RemoveContainer" containerID="85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52"
Sep 18 19:58:14.447779 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.447754    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"]
Sep 18 19:58:14.451712 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.451693    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp"]
Sep 18 19:58:14.451788 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.451768    2560 scope.go:117] "RemoveContainer" containerID="b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f"
Sep 18 19:58:14.452128 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:58:14.452109    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f\": container with ID starting with b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f not found: ID does not exist" containerID="b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f"
Sep 18 19:58:14.452188 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.452137    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f"} err="failed to get container status \"b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f\": rpc error: code = NotFound desc = could not find container \"b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f\": container with ID starting with b334fe8601bf3350445c09a3eebaf3e30f564441eeec0eeb8bef9fbbb563772f not found: ID does not exist"
Sep 18 19:58:14.452188 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.452156    2560 scope.go:117] "RemoveContainer" containerID="71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b"
Sep 18 19:58:14.452415 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:58:14.452397    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b\": container with ID starting with 71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b not found: ID does not exist" containerID="71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b"
Sep 18 19:58:14.452459 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.452421    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b"} err="failed to get container status \"71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b\": rpc error: code = NotFound desc = could not find container \"71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b\": container with ID starting with 71318e61dbfbff6245dd649b76ab7e9a39c70b8689c323e51d92639d26e1ce6b not found: ID does not exist"
Sep 18 19:58:14.452459 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.452436    2560 scope.go:117] "RemoveContainer" containerID="85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52"
Sep 18 19:58:14.452711 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:58:14.452693    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52\": container with ID starting with 85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52 not found: ID does not exist" containerID="85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52"
Sep 18 19:58:14.452747 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.452715    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52"} err="failed to get container status \"85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52\": rpc error: code = NotFound desc = could not find container \"85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52\": container with ID starting with 85b7c2009a1bbeaeb3b6b3109ab13ff8e3f1369e86482c9ddf7fe726d86e6c52 not found: ID does not exist"
Sep 18 19:58:14.695193 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.695134    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:14.746095 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.746060    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:14.928619 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:14.928570    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" path="/var/lib/kubelet/pods/76f5d722-d3bf-4c6d-b25a-34f85e1b59da/volumes"
Sep 18 19:58:15.841660 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:15.841622    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:47.838597 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:47.838549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:48.443669 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:58:48.443623    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:05.623510 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:05.623418    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:10.758714 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.758680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:10.785197 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.785153    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"]
Sep 18 19:59:10.789891 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789827    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container"
Sep 18 19:59:10.789891 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789881    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container"
Sep 18 19:59:10.789891 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789890    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container"
Sep 18 19:59:10.789891 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789896    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789905    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="storage-initializer"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789912    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="storage-initializer"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789923    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="storage-initializer"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789929    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="storage-initializer"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789935    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kube-rbac-proxy"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789941    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kube-rbac-proxy"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789948    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kube-rbac-proxy"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.789953    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kube-rbac-proxy"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.790020    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kube-rbac-proxy"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.790027    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cad6a91-069d-455f-8575-784f23b6a971" containerName="kserve-container"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.790033    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kserve-container"
Sep 18 19:59:10.790140 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.790040    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f5d722-d3bf-4c6d-b25a-34f85e1b59da" containerName="kube-rbac-proxy"
Sep 18 19:59:10.793172 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.793149    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:10.795647 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.795619    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:59:10.795780 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.795663    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 18 19:59:10.795780 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.795682    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 19:59:10.796832 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.796802    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:59:10.796993 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.796944    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:59:10.796993 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.796955    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:59:10.797650 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.797093    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 18 19:59:10.799664 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.799636    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"]
Sep 18 19:59:10.861467 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.861434    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:10.936547 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.936508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6906ec81-957d-4ef5-9503-ea58bd7cf664-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:10.936547 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.936546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:10.936764 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.936650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6906ec81-957d-4ef5-9503-ea58bd7cf664-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:10.936764 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.936683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:10.936764 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:10.936721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b77f\" (UniqueName: \"kubernetes.io/projected/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-api-access-2b77f\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.037982 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.037884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6906ec81-957d-4ef5-9503-ea58bd7cf664-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.037982 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.037934    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.038205 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.037997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6906ec81-957d-4ef5-9503-ea58bd7cf664-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.038205 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.038022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.038205 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.038055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b77f\" (UniqueName: \"kubernetes.io/projected/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-api-access-2b77f\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.038521 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.038493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6906ec81-957d-4ef5-9503-ea58bd7cf664-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.038908 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.038697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.038908 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.038708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.040583 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.040560    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6906ec81-957d-4ef5-9503-ea58bd7cf664-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.046544 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.046516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b77f\" (UniqueName: \"kubernetes.io/projected/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-api-access-2b77f\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.105259 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.105215    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 19:59:11.379071 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.379042    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"]
Sep 18 19:59:11.381468 ip-10-0-142-19 kubenswrapper[2560]: W0918 19:59:11.381422    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6906ec81_957d_4ef5_9503_ea58bd7cf664.slice/crio-9a8b41f9706c1d9f926aa1cdbaea702229775f5dec1e80ce951246cb14c8d281 WatchSource:0}: Error finding container 9a8b41f9706c1d9f926aa1cdbaea702229775f5dec1e80ce951246cb14c8d281: Status 404 returned error can't find the container with id 9a8b41f9706c1d9f926aa1cdbaea702229775f5dec1e80ce951246cb14c8d281
Sep 18 19:59:11.687193 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.687151    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" event={"ID":"6906ec81-957d-4ef5-9503-ea58bd7cf664","Type":"ContainerStarted","Data":"1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34"}
Sep 18 19:59:11.687379 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:11.687230    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" event={"ID":"6906ec81-957d-4ef5-9503-ea58bd7cf664","Type":"ContainerStarted","Data":"9a8b41f9706c1d9f926aa1cdbaea702229775f5dec1e80ce951246cb14c8d281"}
Sep 18 19:59:16.709051 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:16.709017    2560 generic.go:356] "Generic (PLEG): container finished" podID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerID="1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34" exitCode=0
Sep 18 19:59:16.709549 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:16.709098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" event={"ID":"6906ec81-957d-4ef5-9503-ea58bd7cf664","Type":"ContainerDied","Data":"1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34"}
Sep 18 19:59:44.858613 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:44.858576    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843" exitCode=1
Sep 18 19:59:44.859159 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:44.858636    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"}
Sep 18 19:59:44.859159 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:44.858701    2560 scope.go:117] "RemoveContainer" containerID="55278654e56a7404090abefee9f20e741f92a705042f44713aa02e9d2956d603"
Sep 18 19:59:44.859159 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:44.859123    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 19:59:44.859346 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:59:44.859315    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:59:49.212649 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:49.212609    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:59:49.213153 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:49.213136    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 19:59:49.213448 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:59:49.213420    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 19:59:52.046737 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:52.046695    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 19:59:52.047240 ip-10-0-142-19 kubenswrapper[2560]: I0918 19:59:52.047148    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 19:59:52.047358 ip-10-0-142-19 kubenswrapper[2560]: E0918 19:59:52.047338    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:00:05.922766 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:00:05.922727    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:00:05.923262 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:00:05.922954    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:00:16.922375 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:00:16.922338    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:00:16.922900 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:00:16.922612    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:00:17.981599 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:00:17.981563    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:23.100054 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:00:23.100003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:23.200085 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:00:23.200053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:30.922914 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:00:30.922877    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:00:30.923489 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:00:30.923124    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:00:44.923189 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:00:44.922979    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:00:44.923737 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:00:44.923191    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:00:56.922435 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:00:56.922402    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:00:56.923025 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:00:56.922661    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:01:04.275707 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:04.275670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" event={"ID":"6906ec81-957d-4ef5-9503-ea58bd7cf664","Type":"ContainerStarted","Data":"dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4"}
Sep 18 20:01:04.276245 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:04.275751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" event={"ID":"6906ec81-957d-4ef5-9503-ea58bd7cf664","Type":"ContainerStarted","Data":"b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59"}
Sep 18 20:01:04.276245 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:04.275779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 20:01:04.294151 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:04.294079    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" podStartSLOduration=2.765847515 podStartE2EDuration="1m54.294061521s" totalImagesPullingTime="1m46.467312277s" totalInitContainerRuntime="5.060901729s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:04.293068702 +0000 UTC m=+1371.895431885" watchObservedRunningTime="2026-09-18 20:01:04.294061521 +0000 UTC m=+1371.896424746"
Sep 18 20:01:05.280118 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:05.280082    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 20:01:07.922843 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:07.922814    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:01:07.923252 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:01:07.923006    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:01:11.291179 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:11.291141    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 20:01:20.922692 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:20.922660    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:01:20.923131 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:01:20.922828    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:01:35.922638 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:35.922605    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:01:35.923069 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:01:35.922771    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:01:50.923028 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:01:50.922992    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:01:50.923488 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:01:50.923173    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:02:04.922378 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:04.922343    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:02:04.922782 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:02:04.922537    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:02:17.923069 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:17.923029    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:02:17.923478 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:02:17.923251    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:02:30.922881 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:30.922817    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:02:30.923334 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:30.923253    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:02:31.659901 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:31.659864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"}
Sep 18 20:02:31.660097 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:31.660079    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:02:41.956613 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:41.956568    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 20:02:52.964076 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:52.964024    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:02:57.534107 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:57.534067    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"]
Sep 18 20:02:57.534611 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:57.534442    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kserve-container" containerID="cri-o://b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59" gracePeriod=30
Sep 18 20:02:57.534611 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:57.534457    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kube-rbac-proxy" containerID="cri-o://dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4" gracePeriod=30
Sep 18 20:02:57.778378 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:57.778346    2560 generic.go:356] "Generic (PLEG): container finished" podID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerID="dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4" exitCode=2
Sep 18 20:02:57.778526 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:57.778424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" event={"ID":"6906ec81-957d-4ef5-9503-ea58bd7cf664","Type":"ContainerDied","Data":"dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4"}
Sep 18 20:02:58.361446 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:58.361413    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:59.177215 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.177184    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 20:02:59.260434 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.260345    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6906ec81-957d-4ef5-9503-ea58bd7cf664-proxy-tls\") pod \"6906ec81-957d-4ef5-9503-ea58bd7cf664\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") "
Sep 18 20:02:59.260434 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.260383    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-cabundle-cert\") pod \"6906ec81-957d-4ef5-9503-ea58bd7cf664\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") "
Sep 18 20:02:59.260434 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.260416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6906ec81-957d-4ef5-9503-ea58bd7cf664-kserve-provision-location\") pod \"6906ec81-957d-4ef5-9503-ea58bd7cf664\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") "
Sep 18 20:02:59.260434 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.260437    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-rbac-proxy-sar-config\") pod \"6906ec81-957d-4ef5-9503-ea58bd7cf664\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") "
Sep 18 20:02:59.260809 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.260457    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b77f\" (UniqueName: \"kubernetes.io/projected/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-api-access-2b77f\") pod \"6906ec81-957d-4ef5-9503-ea58bd7cf664\" (UID: \"6906ec81-957d-4ef5-9503-ea58bd7cf664\") "
Sep 18 20:02:59.260894 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.260811    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6906ec81-957d-4ef5-9503-ea58bd7cf664" (UID: "6906ec81-957d-4ef5-9503-ea58bd7cf664"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:59.260894 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.260820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6906ec81-957d-4ef5-9503-ea58bd7cf664" (UID: "6906ec81-957d-4ef5-9503-ea58bd7cf664"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:59.262651 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.262614    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6906ec81-957d-4ef5-9503-ea58bd7cf664-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6906ec81-957d-4ef5-9503-ea58bd7cf664" (UID: "6906ec81-957d-4ef5-9503-ea58bd7cf664"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:59.262651 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.262641    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-api-access-2b77f" (OuterVolumeSpecName: "kube-api-access-2b77f") pod "6906ec81-957d-4ef5-9503-ea58bd7cf664" (UID: "6906ec81-957d-4ef5-9503-ea58bd7cf664"). InnerVolumeSpecName "kube-api-access-2b77f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:59.264010 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.263985    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6906ec81-957d-4ef5-9503-ea58bd7cf664-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6906ec81-957d-4ef5-9503-ea58bd7cf664" (UID: "6906ec81-957d-4ef5-9503-ea58bd7cf664"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:59.361786 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.361743    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6906ec81-957d-4ef5-9503-ea58bd7cf664-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:02:59.361786 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.361782    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:02:59.361786 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.361792    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6906ec81-957d-4ef5-9503-ea58bd7cf664-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:02:59.362038 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.361802    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:02:59.362038 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.361817    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b77f\" (UniqueName: \"kubernetes.io/projected/6906ec81-957d-4ef5-9503-ea58bd7cf664-kube-api-access-2b77f\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:02:59.789923 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.789888    2560 generic.go:356] "Generic (PLEG): container finished" podID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerID="b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59" exitCode=0
Sep 18 20:02:59.790092 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.789939    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"
Sep 18 20:02:59.790092 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.789973    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" event={"ID":"6906ec81-957d-4ef5-9503-ea58bd7cf664","Type":"ContainerDied","Data":"b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59"}
Sep 18 20:02:59.790092 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.790036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5" event={"ID":"6906ec81-957d-4ef5-9503-ea58bd7cf664","Type":"ContainerDied","Data":"9a8b41f9706c1d9f926aa1cdbaea702229775f5dec1e80ce951246cb14c8d281"}
Sep 18 20:02:59.790092 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.790062    2560 scope.go:117] "RemoveContainer" containerID="dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4"
Sep 18 20:02:59.798604 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.798581    2560 scope.go:117] "RemoveContainer" containerID="b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59"
Sep 18 20:02:59.806771 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.806749    2560 scope.go:117] "RemoveContainer" containerID="1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34"
Sep 18 20:02:59.811448 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.811418    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"]
Sep 18 20:02:59.815278 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.815252    2560 scope.go:117] "RemoveContainer" containerID="dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4"
Sep 18 20:02:59.815694 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:02:59.815675    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4\": container with ID starting with dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4 not found: ID does not exist" containerID="dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4"
Sep 18 20:02:59.815774 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.815705    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4"} err="failed to get container status \"dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4\": rpc error: code = NotFound desc = could not find container \"dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4\": container with ID starting with dd5b4893371f555c622ed790988474f845554fabbf0d49e2822ec2f1e60ca5c4 not found: ID does not exist"
Sep 18 20:02:59.815774 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.815724    2560 scope.go:117] "RemoveContainer" containerID="b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59"
Sep 18 20:02:59.816051 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:02:59.816032    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59\": container with ID starting with b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59 not found: ID does not exist" containerID="b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59"
Sep 18 20:02:59.816108 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.816058    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59"} err="failed to get container status \"b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59\": rpc error: code = NotFound desc = could not find container \"b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59\": container with ID starting with b121b8e854f9a2f780d9f95d0e271eeec01eee9df9e8c4461e98d837432bfa59 not found: ID does not exist"
Sep 18 20:02:59.816108 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.816075    2560 scope.go:117] "RemoveContainer" containerID="1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34"
Sep 18 20:02:59.816353 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.816331    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5"]
Sep 18 20:02:59.816389 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:02:59.816330    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34\": container with ID starting with 1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34 not found: ID does not exist" containerID="1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34"
Sep 18 20:02:59.816423 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:02:59.816398    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34"} err="failed to get container status \"1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34\": rpc error: code = NotFound desc = could not find container \"1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34\": container with ID starting with 1e906a711507f7f6d05aaf9c75eea24399279970c452ab960aedb8f311ebea34 not found: ID does not exist"
Sep 18 20:03:00.927883 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:00.927826    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" path="/var/lib/kubelet/pods/6906ec81-957d-4ef5-9503-ea58bd7cf664/volumes"
Sep 18 20:03:02.605924 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.605892    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:02.653638 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.653602    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"]
Sep 18 20:03:02.653951 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.653938    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kserve-container"
Sep 18 20:03:02.653999 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.653954    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kserve-container"
Sep 18 20:03:02.653999 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.653972    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kube-rbac-proxy"
Sep 18 20:03:02.653999 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.653978    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kube-rbac-proxy"
Sep 18 20:03:02.653999 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.653989    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="storage-initializer"
Sep 18 20:03:02.653999 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.653995    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="storage-initializer"
Sep 18 20:03:02.654154 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.654052    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kube-rbac-proxy"
Sep 18 20:03:02.654154 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.654062    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6906ec81-957d-4ef5-9503-ea58bd7cf664" containerName="kserve-container"
Sep 18 20:03:02.658721 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.658692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.661463 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.661426    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:03:02.661463 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.661451    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 18 20:03:02.661655 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.661477    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 18 20:03:02.661655 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.661426    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:03:02.661655 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.661483    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:03:02.661655 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.661426    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:03:02.662124 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.662111    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:03:02.665500 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.665472    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:03:02.667176 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.667155    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"]
Sep 18 20:03:02.789763 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.789724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.789963 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.789878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.789963 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.789923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.790078 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.789981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ef7634-ecb3-4a41-84ab-fd316fa69b81-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.790078 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.790003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vn77\" (UniqueName: \"kubernetes.io/projected/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-api-access-6vn77\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.842871 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.842816    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:02.890580 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.890498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.890580 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.890536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.890580 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.890582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ef7634-ecb3-4a41-84ab-fd316fa69b81-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.890879 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.890600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vn77\" (UniqueName: \"kubernetes.io/projected/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-api-access-6vn77\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.890879 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.890637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.891068 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.891048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.891301 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.891267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.891301 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.891289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.893137 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.893117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ef7634-ecb3-4a41-84ab-fd316fa69b81-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.899751 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.899727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vn77\" (UniqueName: \"kubernetes.io/projected/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-api-access-6vn77\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:02.970304 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:02.970260    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:03.242767 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:03.242736    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"]
Sep 18 20:03:03.245142 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:03:03.245108    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ef7634_ecb3_4a41_84ab_fd316fa69b81.slice/crio-66333ad6e84302aadaf01b81c8f22b827a61ebdc427aa26bd8d0105faba29310 WatchSource:0}: Error finding container 66333ad6e84302aadaf01b81c8f22b827a61ebdc427aa26bd8d0105faba29310: Status 404 returned error can't find the container with id 66333ad6e84302aadaf01b81c8f22b827a61ebdc427aa26bd8d0105faba29310
Sep 18 20:03:03.810780 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:03.810735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" event={"ID":"42ef7634-ecb3-4a41-84ab-fd316fa69b81","Type":"ContainerStarted","Data":"5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514"}
Sep 18 20:03:03.811203 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:03.810806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" event={"ID":"42ef7634-ecb3-4a41-84ab-fd316fa69b81","Type":"ContainerStarted","Data":"66333ad6e84302aadaf01b81c8f22b827a61ebdc427aa26bd8d0105faba29310"}
Sep 18 20:03:12.854768 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:12.854729    2560 generic.go:356] "Generic (PLEG): container finished" podID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerID="5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514" exitCode=0
Sep 18 20:03:12.855191 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:12.854803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" event={"ID":"42ef7634-ecb3-4a41-84ab-fd316fa69b81","Type":"ContainerDied","Data":"5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514"}
Sep 18 20:03:13.861475 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:13.861442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" event={"ID":"42ef7634-ecb3-4a41-84ab-fd316fa69b81","Type":"ContainerStarted","Data":"c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad"}
Sep 18 20:03:13.861983 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:13.861503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" event={"ID":"42ef7634-ecb3-4a41-84ab-fd316fa69b81","Type":"ContainerStarted","Data":"d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3"}
Sep 18 20:03:13.861983 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:13.861713    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:13.879476 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:13.879422    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" podStartSLOduration=2.941808768 podStartE2EDuration="11.879403497s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.937594729s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:13.877921803 +0000 UTC m=+1501.480284996" watchObservedRunningTime="2026-09-18 20:03:13.879403497 +0000 UTC m=+1501.481766678"
Sep 18 20:03:14.865880 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:14.865834    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:14.867583 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:14.867546    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:03:15.870739 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:15.870695    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:03:20.875199 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:20.875169    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:20.875778 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:20.875740    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:03:30.877623 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:30.877588    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:33.264048 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:33.264011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:38.418302 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:38.418204    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:38.785222 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:38.785128    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:45.835262 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:45.835224    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"]
Sep 18 20:03:45.835767 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:45.835450    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kserve-container" containerID="cri-o://d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3" gracePeriod=30
Sep 18 20:03:45.835767 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:45.835488    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kube-rbac-proxy" containerID="cri-o://c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad" gracePeriod=30
Sep 18 20:03:45.843984 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:45.843956    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:45.871565 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:45.871525    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 18 20:03:46.016791 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:46.016755    2560 generic.go:356] "Generic (PLEG): container finished" podID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerID="c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad" exitCode=2
Sep 18 20:03:46.016973 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:46.016838    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" event={"ID":"42ef7634-ecb3-4a41-84ab-fd316fa69b81","Type":"ContainerDied","Data":"c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad"}
Sep 18 20:03:47.782326 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.782297    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:47.871273 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.871173    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-rbac-proxy-sar-config\") pod \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") "
Sep 18 20:03:47.871273 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.871230    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ef7634-ecb3-4a41-84ab-fd316fa69b81-proxy-tls\") pod \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") "
Sep 18 20:03:47.871273 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.871254    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vn77\" (UniqueName: \"kubernetes.io/projected/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-api-access-6vn77\") pod \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") "
Sep 18 20:03:47.871567 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.871321    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-cabundle-cert\") pod \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") "
Sep 18 20:03:47.871567 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.871386    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kserve-provision-location\") pod \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\" (UID: \"42ef7634-ecb3-4a41-84ab-fd316fa69b81\") "
Sep 18 20:03:47.871684 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.871612    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42ef7634-ecb3-4a41-84ab-fd316fa69b81" (UID: "42ef7634-ecb3-4a41-84ab-fd316fa69b81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:47.871745 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.871687    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "42ef7634-ecb3-4a41-84ab-fd316fa69b81" (UID: "42ef7634-ecb3-4a41-84ab-fd316fa69b81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:47.872440 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.872414    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42ef7634-ecb3-4a41-84ab-fd316fa69b81" (UID: "42ef7634-ecb3-4a41-84ab-fd316fa69b81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:47.873545 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.873518    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-api-access-6vn77" (OuterVolumeSpecName: "kube-api-access-6vn77") pod "42ef7634-ecb3-4a41-84ab-fd316fa69b81" (UID: "42ef7634-ecb3-4a41-84ab-fd316fa69b81"). InnerVolumeSpecName "kube-api-access-6vn77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:47.873629 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.873607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ef7634-ecb3-4a41-84ab-fd316fa69b81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42ef7634-ecb3-4a41-84ab-fd316fa69b81" (UID: "42ef7634-ecb3-4a41-84ab-fd316fa69b81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:47.972147 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.972100    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:03:47.972147 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.972138    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ef7634-ecb3-4a41-84ab-fd316fa69b81-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:03:47.972147 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.972150    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vn77\" (UniqueName: \"kubernetes.io/projected/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kube-api-access-6vn77\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:03:47.972147 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.972159    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42ef7634-ecb3-4a41-84ab-fd316fa69b81-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:03:47.972418 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:47.972168    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ef7634-ecb3-4a41-84ab-fd316fa69b81-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:03:48.027614 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.027577    2560 generic.go:356] "Generic (PLEG): container finished" podID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerID="d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3" exitCode=0
Sep 18 20:03:48.027788 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.027649    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"
Sep 18 20:03:48.027788 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.027672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" event={"ID":"42ef7634-ecb3-4a41-84ab-fd316fa69b81","Type":"ContainerDied","Data":"d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3"}
Sep 18 20:03:48.027788 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.027754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq" event={"ID":"42ef7634-ecb3-4a41-84ab-fd316fa69b81","Type":"ContainerDied","Data":"66333ad6e84302aadaf01b81c8f22b827a61ebdc427aa26bd8d0105faba29310"}
Sep 18 20:03:48.027974 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.027790    2560 scope.go:117] "RemoveContainer" containerID="c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad"
Sep 18 20:03:48.036676 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.036654    2560 scope.go:117] "RemoveContainer" containerID="d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3"
Sep 18 20:03:48.044193 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.044168    2560 scope.go:117] "RemoveContainer" containerID="5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514"
Sep 18 20:03:48.049737 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.049714    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"]
Sep 18 20:03:48.052869 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.052825    2560 scope.go:117] "RemoveContainer" containerID="c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad"
Sep 18 20:03:48.053210 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:03:48.053188    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad\": container with ID starting with c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad not found: ID does not exist" containerID="c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad"
Sep 18 20:03:48.053282 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.053219    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad"} err="failed to get container status \"c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad\": rpc error: code = NotFound desc = could not find container \"c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad\": container with ID starting with c45cdd1a5ae28fcd072d6dab9091841835a1a5676bb945ad7f9bf5a926072fad not found: ID does not exist"
Sep 18 20:03:48.053282 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.053237    2560 scope.go:117] "RemoveContainer" containerID="d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3"
Sep 18 20:03:48.053538 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:03:48.053520    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3\": container with ID starting with d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3 not found: ID does not exist" containerID="d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3"
Sep 18 20:03:48.053589 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.053550    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3"} err="failed to get container status \"d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3\": rpc error: code = NotFound desc = could not find container \"d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3\": container with ID starting with d95705e0e94ba7705b97aef718bd70025328e32f36beadb793179e889f333df3 not found: ID does not exist"
Sep 18 20:03:48.053589 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.053568    2560 scope.go:117] "RemoveContainer" containerID="5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514"
Sep 18 20:03:48.053804 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:03:48.053787    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514\": container with ID starting with 5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514 not found: ID does not exist" containerID="5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514"
Sep 18 20:03:48.053872 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.053810    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514"} err="failed to get container status \"5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514\": rpc error: code = NotFound desc = could not find container \"5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514\": container with ID starting with 5f770bdf5b87a815b02d750769da6480c36aaebe60f03bf47912a2af29596514 not found: ID does not exist"
Sep 18 20:03:48.063541 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.063513    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq"]
Sep 18 20:03:48.933305 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:48.933268    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" path="/var/lib/kubelet/pods/42ef7634-ecb3-4a41-84ab-fd316fa69b81/volumes"
Sep 18 20:03:50.979355 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:50.979322    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:51.005450 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005415    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"]
Sep 18 20:03:51.005735 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005721    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="storage-initializer"
Sep 18 20:03:51.005782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005736    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="storage-initializer"
Sep 18 20:03:51.005782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005749    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kserve-container"
Sep 18 20:03:51.005782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005755    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kserve-container"
Sep 18 20:03:51.005782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005763    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kube-rbac-proxy"
Sep 18 20:03:51.005782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005768    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kube-rbac-proxy"
Sep 18 20:03:51.006042 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005830    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kserve-container"
Sep 18 20:03:51.006042 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.005840    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ef7634-ecb3-4a41-84ab-fd316fa69b81" containerName="kube-rbac-proxy"
Sep 18 20:03:51.010568 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.010543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.013022 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.012987    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 18 20:03:51.013165 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.013051    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:03:51.013165 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.013072    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:03:51.013165 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.013133    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:03:51.013343 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.013173    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:03:51.013989 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.013971    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:03:51.014089 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.013974    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 18 20:03:51.021647 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.021622    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"]
Sep 18 20:03:51.098320 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.098285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.098320 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.098326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.098556 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.098372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.098556 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.098484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.098556 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.098519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqfrm\" (UniqueName: \"kubernetes.io/projected/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-api-access-mqfrm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.170395 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.170356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:51.199350 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.199302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.199350 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.199349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.199549 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.199453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.199549 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.199510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.199549 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.199529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqfrm\" (UniqueName: \"kubernetes.io/projected/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-api-access-mqfrm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.199928 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.199902    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.200202 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.200180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.200247 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.200180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.202008 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.201984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.207388 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.207361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqfrm\" (UniqueName: \"kubernetes.io/projected/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-api-access-mqfrm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.321049 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.320940    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:51.596678 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:51.596648    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"]
Sep 18 20:03:51.602150 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:03:51.598991    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67efc1d1_0138_4be4_bd2c_fc8b1ba72393.slice/crio-2c28257a7822c6f6b838f0c384769ae8c71eaab8292b9e654af34debf237097e WatchSource:0}: Error finding container 2c28257a7822c6f6b838f0c384769ae8c71eaab8292b9e654af34debf237097e: Status 404 returned error can't find the container with id 2c28257a7822c6f6b838f0c384769ae8c71eaab8292b9e654af34debf237097e
Sep 18 20:03:52.049923 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:52.049877    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" event={"ID":"67efc1d1-0138-4be4-bd2c-fc8b1ba72393","Type":"ContainerStarted","Data":"8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d"}
Sep 18 20:03:52.050287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:52.049964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" event={"ID":"67efc1d1-0138-4be4-bd2c-fc8b1ba72393","Type":"ContainerStarted","Data":"2c28257a7822c6f6b838f0c384769ae8c71eaab8292b9e654af34debf237097e"}
Sep 18 20:03:54.793425 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:54.793387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:57.073880 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:57.073830    2560 generic.go:356] "Generic (PLEG): container finished" podID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerID="8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d" exitCode=0
Sep 18 20:03:57.074264 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:57.073885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" event={"ID":"67efc1d1-0138-4be4-bd2c-fc8b1ba72393","Type":"ContainerDied","Data":"8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d"}
Sep 18 20:03:58.081052 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:58.081017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" event={"ID":"67efc1d1-0138-4be4-bd2c-fc8b1ba72393","Type":"ContainerStarted","Data":"d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6"}
Sep 18 20:03:58.081474 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:58.081083    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" event={"ID":"67efc1d1-0138-4be4-bd2c-fc8b1ba72393","Type":"ContainerStarted","Data":"bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021"}
Sep 18 20:03:58.081474 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:58.081335    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:58.100721 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:58.100666    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" podStartSLOduration=3.380808465 podStartE2EDuration="8.100644854s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.719836389s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:58.09845217 +0000 UTC m=+1545.700815352" watchObservedRunningTime="2026-09-18 20:03:58.100644854 +0000 UTC m=+1545.703008035"
Sep 18 20:03:59.085809 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:59.085773    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:03:59.087228 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:03:59.087196    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:04:00.091687 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:00.091637    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:04:05.097032 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:05.096999    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:04:05.097553 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:05.097524    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:04:10.431139 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:10.431106    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:15.098569 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:15.098536    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:04:15.550149 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:15.550111    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:15.682957 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:15.682924    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:26.562334 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:26.562294    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"]
Sep 18 20:04:26.563235 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:26.562589    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kserve-container" containerID="cri-o://bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021" gracePeriod=30
Sep 18 20:04:26.563235 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:26.562610    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kube-rbac-proxy" containerID="cri-o://d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6" gracePeriod=30
Sep 18 20:04:26.584034 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:26.584003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:27.222272 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:27.222234    2560 generic.go:356] "Generic (PLEG): container finished" podID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerID="d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6" exitCode=2
Sep 18 20:04:27.222486 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:27.222283    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" event={"ID":"67efc1d1-0138-4be4-bd2c-fc8b1ba72393","Type":"ContainerDied","Data":"d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6"}
Sep 18 20:04:28.110452 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.110426    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:04:28.218772 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.218675    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kserve-provision-location\") pod \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") "
Sep 18 20:04:28.218772 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.218717    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-cabundle-cert\") pod \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") "
Sep 18 20:04:28.218772 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.218746    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-rbac-proxy-sar-config\") pod \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") "
Sep 18 20:04:28.219110 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.218780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqfrm\" (UniqueName: \"kubernetes.io/projected/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-api-access-mqfrm\") pod \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") "
Sep 18 20:04:28.219110 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.218817    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-proxy-tls\") pod \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\" (UID: \"67efc1d1-0138-4be4-bd2c-fc8b1ba72393\") "
Sep 18 20:04:28.219110 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.219095    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67efc1d1-0138-4be4-bd2c-fc8b1ba72393" (UID: "67efc1d1-0138-4be4-bd2c-fc8b1ba72393"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:28.219278 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.219200    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67efc1d1-0138-4be4-bd2c-fc8b1ba72393" (UID: "67efc1d1-0138-4be4-bd2c-fc8b1ba72393"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:28.219820 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.219792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67efc1d1-0138-4be4-bd2c-fc8b1ba72393" (UID: "67efc1d1-0138-4be4-bd2c-fc8b1ba72393"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:28.221074 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.221052    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-api-access-mqfrm" (OuterVolumeSpecName: "kube-api-access-mqfrm") pod "67efc1d1-0138-4be4-bd2c-fc8b1ba72393" (UID: "67efc1d1-0138-4be4-bd2c-fc8b1ba72393"). InnerVolumeSpecName "kube-api-access-mqfrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:28.221167 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.221125    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67efc1d1-0138-4be4-bd2c-fc8b1ba72393" (UID: "67efc1d1-0138-4be4-bd2c-fc8b1ba72393"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:28.229174 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.229147    2560 generic.go:356] "Generic (PLEG): container finished" podID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerID="bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021" exitCode=0
Sep 18 20:04:28.229279 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.229201    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"
Sep 18 20:04:28.229279 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.229222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" event={"ID":"67efc1d1-0138-4be4-bd2c-fc8b1ba72393","Type":"ContainerDied","Data":"bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021"}
Sep 18 20:04:28.229356 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.229280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl" event={"ID":"67efc1d1-0138-4be4-bd2c-fc8b1ba72393","Type":"ContainerDied","Data":"2c28257a7822c6f6b838f0c384769ae8c71eaab8292b9e654af34debf237097e"}
Sep 18 20:04:28.229356 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.229309    2560 scope.go:117] "RemoveContainer" containerID="d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6"
Sep 18 20:04:28.238312 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.238292    2560 scope.go:117] "RemoveContainer" containerID="bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021"
Sep 18 20:04:28.246017 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.246000    2560 scope.go:117] "RemoveContainer" containerID="8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d"
Sep 18 20:04:28.251723 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.251693    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"]
Sep 18 20:04:28.254574 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.254551    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl"]
Sep 18 20:04:28.254574 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.254569    2560 scope.go:117] "RemoveContainer" containerID="d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6"
Sep 18 20:04:28.254892 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:04:28.254872    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6\": container with ID starting with d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6 not found: ID does not exist" containerID="d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6"
Sep 18 20:04:28.254960 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.254907    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6"} err="failed to get container status \"d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6\": rpc error: code = NotFound desc = could not find container \"d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6\": container with ID starting with d309169c8fdae8418ec0f5ec998a7e524804ecaa91e4f56fafe425530bf611f6 not found: ID does not exist"
Sep 18 20:04:28.254960 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.254925    2560 scope.go:117] "RemoveContainer" containerID="bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021"
Sep 18 20:04:28.255214 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:04:28.255199    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021\": container with ID starting with bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021 not found: ID does not exist" containerID="bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021"
Sep 18 20:04:28.255310 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.255218    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021"} err="failed to get container status \"bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021\": rpc error: code = NotFound desc = could not find container \"bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021\": container with ID starting with bc82f5c3754029afcd9663609f82ba31fbaa405a1949ae29e37b92650b0e0021 not found: ID does not exist"
Sep 18 20:04:28.255310 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.255232    2560 scope.go:117] "RemoveContainer" containerID="8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d"
Sep 18 20:04:28.255503 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:04:28.255486    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d\": container with ID starting with 8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d not found: ID does not exist" containerID="8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d"
Sep 18 20:04:28.255547 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.255507    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d"} err="failed to get container status \"8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d\": rpc error: code = NotFound desc = could not find container \"8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d\": container with ID starting with 8ddb19e129a2070f638a60f7b19c45c79aef673033e2ef35568f0b08cf519e4d not found: ID does not exist"
Sep 18 20:04:28.319682 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.319639    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:04:28.319682 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.319673    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqfrm\" (UniqueName: \"kubernetes.io/projected/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kube-api-access-mqfrm\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:04:28.319682 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.319683    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:04:28.319682 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.319693    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:04:28.319984 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.319702    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67efc1d1-0138-4be4-bd2c-fc8b1ba72393-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:04:28.928238 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:28.928205    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" path="/var/lib/kubelet/pods/67efc1d1-0138-4be4-bd2c-fc8b1ba72393/volumes"
Sep 18 20:04:31.689231 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.689195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:31.776619 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.776587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:31.858592 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.858556    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"]
Sep 18 20:04:31.858948 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.858933    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kube-rbac-proxy"
Sep 18 20:04:31.858991 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.858949    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kube-rbac-proxy"
Sep 18 20:04:31.858991 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.858960    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kserve-container"
Sep 18 20:04:31.858991 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.858965    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kserve-container"
Sep 18 20:04:31.858991 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.858979    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="storage-initializer"
Sep 18 20:04:31.858991 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.858987    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="storage-initializer"
Sep 18 20:04:31.859174 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.859061    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kserve-container"
Sep 18 20:04:31.859174 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.859072    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67efc1d1-0138-4be4-bd2c-fc8b1ba72393" containerName="kube-rbac-proxy"
Sep 18 20:04:31.863940 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.863916    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:31.867287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.867235    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:04:31.867287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.867277    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 20:04:31.867538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.867519    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:04:31.867589 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.867576    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:04:31.867755 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.867728    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:04:31.868304 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.868281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 20:04:31.868397 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.868303    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:04:31.878207 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.878181    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"]
Sep 18 20:04:31.894044 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.894012    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:31.950576 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.950492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:31.950576 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.950558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:31.950781 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.950594    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:31.950781 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.950637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxwtx\" (UniqueName: \"kubernetes.io/projected/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-api-access-cxwtx\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:31.950781 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.950703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:31.994129 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:31.994095    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:32.052157 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.052119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.052385 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.052172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.052385 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.052209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.052385 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.052239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxwtx\" (UniqueName: \"kubernetes.io/projected/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-api-access-cxwtx\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.052385 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.052275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.052690 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.052665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.053068 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.053043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.053068 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.053060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.054717 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.054700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.060577 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.060547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxwtx\" (UniqueName: \"kubernetes.io/projected/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-api-access-cxwtx\") pod \"isvc-canary-create-predictor-68984b9498-nmsxm\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.174721 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.174679    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:32.445691 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:32.445666    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"]
Sep 18 20:04:32.448027 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:04:32.447995    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81be9c1_61b6_4c9d_8b9e_6951ca3eefe7.slice/crio-eb9acaef9fa515e50be6a6b5acfd75e5ea8a7de794f58dae758828f388f14061 WatchSource:0}: Error finding container eb9acaef9fa515e50be6a6b5acfd75e5ea8a7de794f58dae758828f388f14061: Status 404 returned error can't find the container with id eb9acaef9fa515e50be6a6b5acfd75e5ea8a7de794f58dae758828f388f14061
Sep 18 20:04:33.253831 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:33.253795    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" event={"ID":"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7","Type":"ContainerStarted","Data":"f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873"}
Sep 18 20:04:33.254234 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:33.253872    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" event={"ID":"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7","Type":"ContainerStarted","Data":"eb9acaef9fa515e50be6a6b5acfd75e5ea8a7de794f58dae758828f388f14061"}
Sep 18 20:04:37.272043 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:37.272009    2560 generic.go:356] "Generic (PLEG): container finished" podID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerID="f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873" exitCode=0
Sep 18 20:04:37.272426 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:37.272080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" event={"ID":"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7","Type":"ContainerDied","Data":"f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873"}
Sep 18 20:04:51.356037 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.356003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" event={"ID":"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7","Type":"ContainerStarted","Data":"fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137"}
Sep 18 20:04:51.356486 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.356076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" event={"ID":"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7","Type":"ContainerStarted","Data":"78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293"}
Sep 18 20:04:51.356486 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.356390    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:51.356486 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.356454    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:51.357703 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.357677    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:04:51.357890 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.357868    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b" exitCode=1
Sep 18 20:04:51.357977 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.357869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"}
Sep 18 20:04:51.357977 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.357962    2560 scope.go:117] "RemoveContainer" containerID="cae9d06e40d87e6c9b0758cc3704c5f73e46b758157abb8a5e45fed22de4a843"
Sep 18 20:04:51.358215 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.358200    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:04:51.358392 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:04:51.358372    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:04:51.375409 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:51.375359    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podStartSLOduration=3.516954911 podStartE2EDuration="20.375342804s" totalImagesPullingTime="13.036088279s" totalInitContainerRuntime="3.822299614s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:51.372304022 +0000 UTC m=+1598.974667205" watchObservedRunningTime="2026-09-18 20:04:51.375342804 +0000 UTC m=+1598.977705985"
Sep 18 20:04:52.046410 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:52.046368    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:04:52.364703 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:52.364612    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:04:52.365123 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:52.364764    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:04:52.365123 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:04:52.364874    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:04:52.664601 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:52.664564    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:04:53.369071 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:53.369045    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:04:53.369432 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:04:53.369262    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:04:57.369447 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:57.369419    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:04:57.369963 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:04:57.369934    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:05:04.922869 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:04.922820    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:05:04.923332 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:05:04.923039    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:05:07.370267 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:07.370220    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:05:15.655934 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:15.655896    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:16.922035 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:16.922003    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:05:16.922418 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:05:16.922184    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:05:17.369912 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:17.369807    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:05:20.772837 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.772798    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:20.798129 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.798098    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"]
Sep 18 20:05:20.801935 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.801915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.804237 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.804212    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 18 20:05:20.804379 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.804268    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 18 20:05:20.811538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.811508    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"]
Sep 18 20:05:20.872931 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.872891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr48w\" (UniqueName: \"kubernetes.io/projected/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-api-access-rr48w\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.873111 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.873015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.873111 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.873045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.873111 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.873089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.873250 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.873114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.974023 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.973983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.974023 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.974025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.974301 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.974093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr48w\" (UniqueName: \"kubernetes.io/projected/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-api-access-rr48w\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.974301 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:05:20.974164    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 18 20:05:20.974301 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.974177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.974301 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.974204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.974301 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:05:20.974235    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls podName:bac93705-b9e3-4dbe-8eeb-08ca72f458e0 nodeName:}" failed. No retries permitted until 2026-09-18 20:05:21.474215534 +0000 UTC m=+1629.076578696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls") pod "isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" (UID: "bac93705-b9e3-4dbe-8eeb-08ca72f458e0") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 18 20:05:20.974618 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.974596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.974789 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.974772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.974830 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.974772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:20.982474 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:20.982439    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr48w\" (UniqueName: \"kubernetes.io/projected/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-api-access-rr48w\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:21.477631 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:21.477577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:21.480361 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:21.480310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:21.712945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:21.712908    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:21.832525 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:21.832482    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:21.989225 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:21.989183    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"]
Sep 18 20:05:21.992633 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:05:21.992601    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac93705_b9e3_4dbe_8eeb_08ca72f458e0.slice/crio-d0a2a3b62639f2304d5672aa10a68fa6fa3aebe4ae59ba5a5178b9f23ce86dcb WatchSource:0}: Error finding container d0a2a3b62639f2304d5672aa10a68fa6fa3aebe4ae59ba5a5178b9f23ce86dcb: Status 404 returned error can't find the container with id d0a2a3b62639f2304d5672aa10a68fa6fa3aebe4ae59ba5a5178b9f23ce86dcb
Sep 18 20:05:22.502926 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:22.502887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" event={"ID":"bac93705-b9e3-4dbe-8eeb-08ca72f458e0","Type":"ContainerStarted","Data":"01ea374c7cd28873623cc86b6b5c0acaff45f835254f4442dc8ce42c96939f71"}
Sep 18 20:05:22.503132 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:22.502955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" event={"ID":"bac93705-b9e3-4dbe-8eeb-08ca72f458e0","Type":"ContainerStarted","Data":"d0a2a3b62639f2304d5672aa10a68fa6fa3aebe4ae59ba5a5178b9f23ce86dcb"}
Sep 18 20:05:26.522800 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:26.522765    2560 generic.go:356] "Generic (PLEG): container finished" podID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerID="01ea374c7cd28873623cc86b6b5c0acaff45f835254f4442dc8ce42c96939f71" exitCode=0
Sep 18 20:05:26.523221 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:26.522840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" event={"ID":"bac93705-b9e3-4dbe-8eeb-08ca72f458e0","Type":"ContainerDied","Data":"01ea374c7cd28873623cc86b6b5c0acaff45f835254f4442dc8ce42c96939f71"}
Sep 18 20:05:27.370340 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:27.370294    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:05:27.529195 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:27.529159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" event={"ID":"bac93705-b9e3-4dbe-8eeb-08ca72f458e0","Type":"ContainerStarted","Data":"3abe81757aa5680a42fe769831113699c8f7a346cee93db2fbc46a6328d30a22"}
Sep 18 20:05:27.529587 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:27.529224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" event={"ID":"bac93705-b9e3-4dbe-8eeb-08ca72f458e0","Type":"ContainerStarted","Data":"509ada86abe34596b32c070efa2191a230ba3d734063763da83f722a901211c7"}
Sep 18 20:05:27.529587 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:27.529491    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:27.529667 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:27.529618    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:27.530969 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:27.530948    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:05:27.546629 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:27.546561    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podStartSLOduration=3.9064523639999997 podStartE2EDuration="7.546541415s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.640089051s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:27.545553026 +0000 UTC m=+1635.147916207" watchObservedRunningTime="2026-09-18 20:05:27.546541415 +0000 UTC m=+1635.148904599"
Sep 18 20:05:28.534476 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:28.534428    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:05:28.922690 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:28.922657    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:05:28.922916 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:05:28.922841    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:05:33.539763 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:33.539731    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:05:33.540268 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:33.540242    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:05:37.370904 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:37.370838    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:05:40.927945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:40.927912    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:05:40.928381 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:05:40.928156    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:05:43.540420 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:43.540372    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:05:47.370320 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:47.370277    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:05:53.541218 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:53.541172    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:05:53.922447 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:53.922419    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:05:53.922608 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:05:53.922592    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:05:54.128932 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:54.128890    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"]
Sep 18 20:05:54.129230 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:54.129193    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" containerID="cri-o://78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293" gracePeriod=30
Sep 18 20:05:54.129395 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:54.129281    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy" containerID="cri-o://fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137" gracePeriod=30
Sep 18 20:05:54.129459 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:54.129399    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:05:54.656599 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:54.656568    2560 generic.go:356] "Generic (PLEG): container finished" podID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerID="fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137" exitCode=2
Sep 18 20:05:54.657021 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:54.656650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" event={"ID":"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7","Type":"ContainerDied","Data":"fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137"}
Sep 18 20:05:57.365052 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:05:57.365006    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 18 20:06:01.813981 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:01.813943    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:01.875361 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:01.875327    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:02.365220 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:02.365184    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 18 20:06:03.541126 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:03.541082    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:06:04.130096 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:04.130048    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:06:06.931528 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:06.931496    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:06.999632 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:06.999601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:07.042835 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.042800    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"]
Sep 18 20:06:07.046615 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.046593    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.048790 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.048771    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 20:06:07.048920 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.048811    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:06:07.055689 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.055656    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"]
Sep 18 20:06:07.067609 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.067575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:07.068287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.068258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfp5m\" (UniqueName: \"kubernetes.io/projected/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-api-access-sfp5m\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.068438 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.068302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.068438 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.068345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.068438 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.068411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.068581 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.068483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480c4dc0-83af-4442-afdd-bf77a67a58ac-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.169293 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.169263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.169469 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.169342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.169469 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.169395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480c4dc0-83af-4442-afdd-bf77a67a58ac-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.169579 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.169470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfp5m\" (UniqueName: \"kubernetes.io/projected/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-api-access-sfp5m\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.169579 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.169497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.169661 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:07.169613    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 18 20:06:07.169704 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:07.169681    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls podName:480c4dc0-83af-4442-afdd-bf77a67a58ac nodeName:}" failed. No retries permitted until 2026-09-18 20:06:07.6696597 +0000 UTC m=+1675.272022863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls") pod "isvc-canary-promote-predictor-cb75746f8-5lwjh" (UID: "480c4dc0-83af-4442-afdd-bf77a67a58ac") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 18 20:06:07.169807 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.169785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480c4dc0-83af-4442-afdd-bf77a67a58ac-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.170035 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.170019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.170072 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.170020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.177325 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.177298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfp5m\" (UniqueName: \"kubernetes.io/projected/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-api-access-sfp5m\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.235686 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.235609    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:07.366064 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.366018    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 18 20:06:07.366253 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.366148    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:06:07.674985 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.674950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.677450 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.677427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-5lwjh\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:07.922126 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.922085    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:06:07.922323 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:07.922295    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:06:07.958382 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:07.958285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:08.229305 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:08.229278    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"]
Sep 18 20:06:08.723627 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:08.723584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" event={"ID":"480c4dc0-83af-4442-afdd-bf77a67a58ac","Type":"ContainerStarted","Data":"351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059"}
Sep 18 20:06:08.723813 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:08.723697    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" event={"ID":"480c4dc0-83af-4442-afdd-bf77a67a58ac","Type":"ContainerStarted","Data":"3c28c15f59c83be9c65a98a96668772bee675c760b19c88ca767ed5d70e8cc3a"}
Sep 18 20:06:12.366098 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:12.365995    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 18 20:06:12.743920 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:12.743886    2560 generic.go:356] "Generic (PLEG): container finished" podID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerID="351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059" exitCode=0
Sep 18 20:06:12.744106 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:12.743961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" event={"ID":"480c4dc0-83af-4442-afdd-bf77a67a58ac","Type":"ContainerDied","Data":"351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059"}
Sep 18 20:06:13.540227 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:13.540178    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:06:13.756310 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:13.756273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" event={"ID":"480c4dc0-83af-4442-afdd-bf77a67a58ac","Type":"ContainerStarted","Data":"ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392"}
Sep 18 20:06:13.756508 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:13.756376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" event={"ID":"480c4dc0-83af-4442-afdd-bf77a67a58ac","Type":"ContainerStarted","Data":"b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c"}
Sep 18 20:06:13.756608 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:13.756588    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:13.776120 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:13.776067    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podStartSLOduration=3.001537015 podStartE2EDuration="6.776049287s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.774512272s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:13.774976653 +0000 UTC m=+1681.377339837" watchObservedRunningTime="2026-09-18 20:06:13.776049287 +0000 UTC m=+1681.378412469"
Sep 18 20:06:14.131083 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:14.131052    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:06:14.761083 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:14.761052    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:14.762620 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:14.762590    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:06:15.765094 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:15.765057    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:06:17.365038 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:17.364986    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 18 20:06:19.922369 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:19.922328    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:06:19.922753 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:19.922520    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:06:20.769865 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:20.769821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:06:20.770460 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:20.770430    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:06:22.365952 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:22.365903    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 18 20:06:23.540476 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:23.540438    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:06:24.783740 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.783709    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:06:24.807728 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.807685    2560 generic.go:356] "Generic (PLEG): container finished" podID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerID="78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293" exitCode=137
Sep 18 20:06:24.807910 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.807745    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"
Sep 18 20:06:24.807910 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.807757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" event={"ID":"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7","Type":"ContainerDied","Data":"78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293"}
Sep 18 20:06:24.807910 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.807821    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm" event={"ID":"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7","Type":"ContainerDied","Data":"eb9acaef9fa515e50be6a6b5acfd75e5ea8a7de794f58dae758828f388f14061"}
Sep 18 20:06:24.807910 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.807865    2560 scope.go:117] "RemoveContainer" containerID="fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137"
Sep 18 20:06:24.816883 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.816839    2560 scope.go:117] "RemoveContainer" containerID="78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293"
Sep 18 20:06:24.826249 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.826226    2560 scope.go:117] "RemoveContainer" containerID="f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873"
Sep 18 20:06:24.828455 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828434    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kserve-provision-location\") pod \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") "
Sep 18 20:06:24.828547 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828465    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-rbac-proxy-sar-config\") pod \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") "
Sep 18 20:06:24.828547 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828504    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-cabundle-cert\") pod \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") "
Sep 18 20:06:24.828547 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxwtx\" (UniqueName: \"kubernetes.io/projected/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-api-access-cxwtx\") pod \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") "
Sep 18 20:06:24.828698 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828569    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-proxy-tls\") pod \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\" (UID: \"d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7\") "
Sep 18 20:06:24.828756 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828706    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" (UID: "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:24.828922 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828897    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:24.828922 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828896    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" (UID: "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:24.829058 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.828954    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" (UID: "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:24.830566 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.830538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" (UID: "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:24.830644 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.830598    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-api-access-cxwtx" (OuterVolumeSpecName: "kube-api-access-cxwtx") pod "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" (UID: "d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7"). InnerVolumeSpecName "kube-api-access-cxwtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:24.834137 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.834116    2560 scope.go:117] "RemoveContainer" containerID="fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137"
Sep 18 20:06:24.834396 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:24.834379    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137\": container with ID starting with fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137 not found: ID does not exist" containerID="fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137"
Sep 18 20:06:24.834448 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.834405    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137"} err="failed to get container status \"fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137\": rpc error: code = NotFound desc = could not find container \"fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137\": container with ID starting with fc616309003e2892c85f65f3a9422a17b22649d3f9d440058fb4688e2f312137 not found: ID does not exist"
Sep 18 20:06:24.834448 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.834423    2560 scope.go:117] "RemoveContainer" containerID="78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293"
Sep 18 20:06:24.834723 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:24.834695    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293\": container with ID starting with 78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293 not found: ID does not exist" containerID="78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293"
Sep 18 20:06:24.834772 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.834730    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293"} err="failed to get container status \"78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293\": rpc error: code = NotFound desc = could not find container \"78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293\": container with ID starting with 78de0d7750f26451442c4226bdce72fbadd6c48ae6c0f6b076c8f2457133b293 not found: ID does not exist"
Sep 18 20:06:24.834772 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.834746    2560 scope.go:117] "RemoveContainer" containerID="f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873"
Sep 18 20:06:24.835049 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:24.835033    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873\": container with ID starting with f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873 not found: ID does not exist" containerID="f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873"
Sep 18 20:06:24.835096 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.835053    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873"} err="failed to get container status \"f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873\": rpc error: code = NotFound desc = could not find container \"f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873\": container with ID starting with f8858dca09052336d3427a8e0b589c708cf9a9fc6d20bc185b86e53e5bf32873 not found: ID does not exist"
Sep 18 20:06:24.929369 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.929338    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:24.929369 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.929366    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:24.929584 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.929385    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:24.929584 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:24.929419    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxwtx\" (UniqueName: \"kubernetes.io/projected/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7-kube-api-access-cxwtx\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:25.123690 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:25.123653    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"]
Sep 18 20:06:25.129294 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:25.129269    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm"]
Sep 18 20:06:26.928017 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:26.927974    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" path="/var/lib/kubelet/pods/d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7/volumes"
Sep 18 20:06:30.770330 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:30.770290    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:06:32.924780 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:32.924750    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:06:32.925192 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:32.924926    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:06:33.540399 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:33.540357    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:06:40.771091 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:40.771004    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:06:43.541820 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:43.541790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:06:47.922735 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:47.922703    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:06:47.923142 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:06:47.922840    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:06:50.771292 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:50.771247    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:06:50.945586 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:50.945551    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"]
Sep 18 20:06:50.945859 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:50.945799    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" containerID="cri-o://509ada86abe34596b32c070efa2191a230ba3d734063763da83f722a901211c7" gracePeriod=30
Sep 18 20:06:50.945933 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:50.945837    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kube-rbac-proxy" containerID="cri-o://3abe81757aa5680a42fe769831113699c8f7a346cee93db2fbc46a6328d30a22" gracePeriod=30
Sep 18 20:06:50.964209 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:50.964184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:51.931632 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:51.931601    2560 generic.go:356] "Generic (PLEG): container finished" podID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerID="3abe81757aa5680a42fe769831113699c8f7a346cee93db2fbc46a6328d30a22" exitCode=2
Sep 18 20:06:51.932023 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:51.931681    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" event={"ID":"bac93705-b9e3-4dbe-8eeb-08ca72f458e0","Type":"ContainerDied","Data":"3abe81757aa5680a42fe769831113699c8f7a346cee93db2fbc46a6328d30a22"}
Sep 18 20:06:53.535419 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:53.535371    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 18 20:06:53.541038 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:53.541000    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:06:55.952314 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:55.952278    2560 generic.go:356] "Generic (PLEG): container finished" podID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerID="509ada86abe34596b32c070efa2191a230ba3d734063763da83f722a901211c7" exitCode=0
Sep 18 20:06:55.952314 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:55.952323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" event={"ID":"bac93705-b9e3-4dbe-8eeb-08ca72f458e0","Type":"ContainerDied","Data":"509ada86abe34596b32c070efa2191a230ba3d734063763da83f722a901211c7"}
Sep 18 20:06:55.995142 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:55.995115    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:06:56.000145 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.000119    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-cabundle-cert\") pod \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") "
Sep 18 20:06:56.000286 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.000163    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-rbac-proxy-sar-config\") pod \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") "
Sep 18 20:06:56.000286 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.000207    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr48w\" (UniqueName: \"kubernetes.io/projected/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-api-access-rr48w\") pod \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") "
Sep 18 20:06:56.000286 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.000238    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls\") pod \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") "
Sep 18 20:06:56.000286 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.000257    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kserve-provision-location\") pod \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\" (UID: \"bac93705-b9e3-4dbe-8eeb-08ca72f458e0\") "
Sep 18 20:06:56.000519 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.000488    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bac93705-b9e3-4dbe-8eeb-08ca72f458e0" (UID: "bac93705-b9e3-4dbe-8eeb-08ca72f458e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:56.000665 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.000638    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bac93705-b9e3-4dbe-8eeb-08ca72f458e0" (UID: "bac93705-b9e3-4dbe-8eeb-08ca72f458e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:56.000757 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.000643    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bac93705-b9e3-4dbe-8eeb-08ca72f458e0" (UID: "bac93705-b9e3-4dbe-8eeb-08ca72f458e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:56.002418 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.002387    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-api-access-rr48w" (OuterVolumeSpecName: "kube-api-access-rr48w") pod "bac93705-b9e3-4dbe-8eeb-08ca72f458e0" (UID: "bac93705-b9e3-4dbe-8eeb-08ca72f458e0"). InnerVolumeSpecName "kube-api-access-rr48w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:56.002525 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.002437    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bac93705-b9e3-4dbe-8eeb-08ca72f458e0" (UID: "bac93705-b9e3-4dbe-8eeb-08ca72f458e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:56.084282 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.084188    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:56.101657 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.101624    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr48w\" (UniqueName: \"kubernetes.io/projected/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-api-access-rr48w\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:56.101657 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.101657    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:56.101894 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.101674    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:56.101894 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.101686    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:56.101894 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.101697    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bac93705-b9e3-4dbe-8eeb-08ca72f458e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:06:56.402340 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.402299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:56.959084 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.959046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt" event={"ID":"bac93705-b9e3-4dbe-8eeb-08ca72f458e0","Type":"ContainerDied","Data":"d0a2a3b62639f2304d5672aa10a68fa6fa3aebe4ae59ba5a5178b9f23ce86dcb"}
Sep 18 20:06:56.959488 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.959120    2560 scope.go:117] "RemoveContainer" containerID="3abe81757aa5680a42fe769831113699c8f7a346cee93db2fbc46a6328d30a22"
Sep 18 20:06:56.959488 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.959071    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"
Sep 18 20:06:56.967565 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.967547    2560 scope.go:117] "RemoveContainer" containerID="509ada86abe34596b32c070efa2191a230ba3d734063763da83f722a901211c7"
Sep 18 20:06:56.976001 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.975975    2560 scope.go:117] "RemoveContainer" containerID="01ea374c7cd28873623cc86b6b5c0acaff45f835254f4442dc8ce42c96939f71"
Sep 18 20:06:56.978265 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.978242    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"]
Sep 18 20:06:56.981949 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:56.981926    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt"]
Sep 18 20:06:58.928639 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:06:58.928604    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" path="/var/lib/kubelet/pods/bac93705-b9e3-4dbe-8eeb-08ca72f458e0/volumes"
Sep 18 20:07:00.770585 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:00.770548    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:07:02.924836 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:02.924799    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:07:02.925272 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:07:02.925011    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:07:10.771022 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:10.770978    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:07:15.922785 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:15.922754    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:07:15.923213 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:07:15.922961    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:07:20.771028 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:20.770987    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:07:27.922241 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:27.922211    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:07:27.922606 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:07:27.922391    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:07:28.556311 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:28.556276    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"]
Sep 18 20:07:28.556558 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:28.556535    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container" containerID="cri-o://b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c" gracePeriod=30
Sep 18 20:07:28.556636 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:28.556579    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kube-rbac-proxy" containerID="cri-o://ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392" gracePeriod=30
Sep 18 20:07:28.557875 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:28.557831    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:07:29.110236 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:29.110199    2560 generic.go:356] "Generic (PLEG): container finished" podID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerID="ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392" exitCode=2
Sep 18 20:07:29.110619 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:29.110249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" event={"ID":"480c4dc0-83af-4442-afdd-bf77a67a58ac","Type":"ContainerDied","Data":"ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392"}
Sep 18 20:07:30.766134 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:30.766083    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 18 20:07:33.502253 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.502225    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:07:33.609422 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.609326    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfp5m\" (UniqueName: \"kubernetes.io/projected/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-api-access-sfp5m\") pod \"480c4dc0-83af-4442-afdd-bf77a67a58ac\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") "
Sep 18 20:07:33.609422 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.609373    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-rbac-proxy-sar-config\") pod \"480c4dc0-83af-4442-afdd-bf77a67a58ac\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") "
Sep 18 20:07:33.609619 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.609426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls\") pod \"480c4dc0-83af-4442-afdd-bf77a67a58ac\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") "
Sep 18 20:07:33.609619 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.609601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480c4dc0-83af-4442-afdd-bf77a67a58ac-kserve-provision-location\") pod \"480c4dc0-83af-4442-afdd-bf77a67a58ac\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") "
Sep 18 20:07:33.609710 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.609670    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-cabundle-cert\") pod \"480c4dc0-83af-4442-afdd-bf77a67a58ac\" (UID: \"480c4dc0-83af-4442-afdd-bf77a67a58ac\") "
Sep 18 20:07:33.609907 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.609871    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "480c4dc0-83af-4442-afdd-bf77a67a58ac" (UID: "480c4dc0-83af-4442-afdd-bf77a67a58ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:33.610042 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.609950    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480c4dc0-83af-4442-afdd-bf77a67a58ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "480c4dc0-83af-4442-afdd-bf77a67a58ac" (UID: "480c4dc0-83af-4442-afdd-bf77a67a58ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:33.610042 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.610028    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "480c4dc0-83af-4442-afdd-bf77a67a58ac" (UID: "480c4dc0-83af-4442-afdd-bf77a67a58ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:33.611544 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.611518    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-api-access-sfp5m" (OuterVolumeSpecName: "kube-api-access-sfp5m") pod "480c4dc0-83af-4442-afdd-bf77a67a58ac" (UID: "480c4dc0-83af-4442-afdd-bf77a67a58ac"). InnerVolumeSpecName "kube-api-access-sfp5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:33.611631 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.611610    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "480c4dc0-83af-4442-afdd-bf77a67a58ac" (UID: "480c4dc0-83af-4442-afdd-bf77a67a58ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:33.710625 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.710582    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/480c4dc0-83af-4442-afdd-bf77a67a58ac-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:07:33.710625 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.710618    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:07:33.710625 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.710630    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfp5m\" (UniqueName: \"kubernetes.io/projected/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-api-access-sfp5m\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:07:33.710894 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.710639    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/480c4dc0-83af-4442-afdd-bf77a67a58ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:07:33.710894 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:33.710648    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/480c4dc0-83af-4442-afdd-bf77a67a58ac-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:07:34.138608 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.138577    2560 generic.go:356] "Generic (PLEG): container finished" podID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerID="b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c" exitCode=0
Sep 18 20:07:34.138785 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.138638    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"
Sep 18 20:07:34.138785 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.138664    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" event={"ID":"480c4dc0-83af-4442-afdd-bf77a67a58ac","Type":"ContainerDied","Data":"b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c"}
Sep 18 20:07:34.138785 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.138724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh" event={"ID":"480c4dc0-83af-4442-afdd-bf77a67a58ac","Type":"ContainerDied","Data":"3c28c15f59c83be9c65a98a96668772bee675c760b19c88ca767ed5d70e8cc3a"}
Sep 18 20:07:34.138785 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.138753    2560 scope.go:117] "RemoveContainer" containerID="ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392"
Sep 18 20:07:34.148070 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.148054    2560 scope.go:117] "RemoveContainer" containerID="b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c"
Sep 18 20:07:34.155374 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.155343    2560 scope.go:117] "RemoveContainer" containerID="351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059"
Sep 18 20:07:34.160600 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.160562    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"]
Sep 18 20:07:34.163867 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.163778    2560 scope.go:117] "RemoveContainer" containerID="ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392"
Sep 18 20:07:34.164330 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:07:34.164251    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392\": container with ID starting with ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392 not found: ID does not exist" containerID="ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392"
Sep 18 20:07:34.164330 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.164298    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392"} err="failed to get container status \"ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392\": rpc error: code = NotFound desc = could not find container \"ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392\": container with ID starting with ee8ef36376db82eb35321d592a44955680fb3c05defdede6ce9471bc15452392 not found: ID does not exist"
Sep 18 20:07:34.164330 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.164323    2560 scope.go:117] "RemoveContainer" containerID="b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c"
Sep 18 20:07:34.164670 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:07:34.164641    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c\": container with ID starting with b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c not found: ID does not exist" containerID="b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c"
Sep 18 20:07:34.164747 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.164680    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c"} err="failed to get container status \"b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c\": rpc error: code = NotFound desc = could not find container \"b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c\": container with ID starting with b9899755c294da9b18904bebb4427598d5b82cf23712729537a2fb0ce542cf1c not found: ID does not exist"
Sep 18 20:07:34.164747 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.164706    2560 scope.go:117] "RemoveContainer" containerID="351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059"
Sep 18 20:07:34.165086 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:07:34.165060    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059\": container with ID starting with 351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059 not found: ID does not exist" containerID="351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059"
Sep 18 20:07:34.165163 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.165092    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059"} err="failed to get container status \"351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059\": rpc error: code = NotFound desc = could not find container \"351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059\": container with ID starting with 351574e9d0818e7a17539f3ce98e6a9cdad415854bbf536acaa0f9e4af384059 not found: ID does not exist"
Sep 18 20:07:34.166112 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.166095    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh"]
Sep 18 20:07:34.927743 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:34.927707    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" path="/var/lib/kubelet/pods/480c4dc0-83af-4442-afdd-bf77a67a58ac/volumes"
Sep 18 20:07:37.029773 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.029740    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:37.094240 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094210    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"]
Sep 18 20:07:37.094566 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094552    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container"
Sep 18 20:07:37.094611 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094569    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container"
Sep 18 20:07:37.094611 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094583    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container"
Sep 18 20:07:37.094611 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094592    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container"
Sep 18 20:07:37.094611 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094604    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kube-rbac-proxy"
Sep 18 20:07:37.094611 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094610    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kube-rbac-proxy"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094617    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="storage-initializer"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094623    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="storage-initializer"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094635    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094640    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094647    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094652    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094658    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="storage-initializer"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094663    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="storage-initializer"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094671    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="storage-initializer"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094680    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="storage-initializer"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094691    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kube-rbac-proxy"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094696    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kube-rbac-proxy"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094742    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kserve-container"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094751    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81be9c1-61b6-4c9d-8b9e-6951ca3eefe7" containerName="kube-rbac-proxy"
Sep 18 20:07:37.094759 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094766    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kserve-container"
Sep 18 20:07:37.095204 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094771    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kserve-container"
Sep 18 20:07:37.095204 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094777    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="480c4dc0-83af-4442-afdd-bf77a67a58ac" containerName="kube-rbac-proxy"
Sep 18 20:07:37.095204 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.094783    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac93705-b9e3-4dbe-8eeb-08ca72f458e0" containerName="kube-rbac-proxy"
Sep 18 20:07:37.099556 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.099535    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.103130 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.102927    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:07:37.103130 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.102960    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:07:37.103130 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.102982    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:07:37.103130 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.102989    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:07:37.103130 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.102999    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 20:07:37.103130 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.103019    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:07:37.103130 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.102982    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:07:37.107645 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.107596    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"]
Sep 18 20:07:37.239551 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.239502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.239551 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.239552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpzg9\" (UniqueName: \"kubernetes.io/projected/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-api-access-qpzg9\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.239812 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.239580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.239812 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.239659    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.239812 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.239725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.340149 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.340056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.340149 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.340135    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.340369 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.340181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.340369 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.340346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpzg9\" (UniqueName: \"kubernetes.io/projected/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-api-access-qpzg9\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.340476 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.340397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.340806 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.340778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.340963 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.340912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.340963 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.340957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.343149 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.343118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.347874 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.347828    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpzg9\" (UniqueName: \"kubernetes.io/projected/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-api-access-qpzg9\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-sn66z\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.411684 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.411640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:37.686789 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:37.686752    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"]
Sep 18 20:07:37.690188 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:07:37.690160    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4c0fd4_83ab_45d8_aba6_7e371d973f56.slice/crio-f07142703ace68f47caa8933fb0912d5b4aba98305bd096f5392704861b65f8b WatchSource:0}: Error finding container f07142703ace68f47caa8933fb0912d5b4aba98305bd096f5392704861b65f8b: Status 404 returned error can't find the container with id f07142703ace68f47caa8933fb0912d5b4aba98305bd096f5392704861b65f8b
Sep 18 20:07:38.160039 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:38.159995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" event={"ID":"8e4c0fd4-83ab-45d8-aba6-7e371d973f56","Type":"ContainerStarted","Data":"6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d"}
Sep 18 20:07:38.160480 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:38.160080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" event={"ID":"8e4c0fd4-83ab-45d8-aba6-7e371d973f56","Type":"ContainerStarted","Data":"f07142703ace68f47caa8933fb0912d5b4aba98305bd096f5392704861b65f8b"}
Sep 18 20:07:41.922156 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:41.922114    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:07:41.922541 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:07:41.922264    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:07:42.178206 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:42.178173    2560 generic.go:356] "Generic (PLEG): container finished" podID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerID="6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d" exitCode=0
Sep 18 20:07:42.178384 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:42.178233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" event={"ID":"8e4c0fd4-83ab-45d8-aba6-7e371d973f56","Type":"ContainerDied","Data":"6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d"}
Sep 18 20:07:43.184997 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:43.184961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" event={"ID":"8e4c0fd4-83ab-45d8-aba6-7e371d973f56","Type":"ContainerStarted","Data":"d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b"}
Sep 18 20:07:43.185445 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:43.185028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" event={"ID":"8e4c0fd4-83ab-45d8-aba6-7e371d973f56","Type":"ContainerStarted","Data":"b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3"}
Sep 18 20:07:43.185445 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:43.185228    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:43.203474 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:43.203418    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podStartSLOduration=2.216854674 podStartE2EDuration="6.20340239s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.986547716s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:43.202156856 +0000 UTC m=+1770.804520039" watchObservedRunningTime="2026-09-18 20:07:43.20340239 +0000 UTC m=+1770.805765563"
Sep 18 20:07:44.189176 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:44.189142    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:44.190673 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:44.190642    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:07:45.193618 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:45.193578    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:07:50.198909 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:50.198875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:07:50.199490 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:50.199458    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:07:53.922899 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:07:53.922831    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:07:53.923283 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:07:53.923076    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:08:00.199716 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:00.199671    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:08:04.922761 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:04.922723    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:08:04.923166 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:08:04.922920    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:08:10.199635 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:10.199589    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:08:18.922670 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:18.922634    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:08:18.923234 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:08:18.922828    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:08:20.199775 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:20.199734    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:08:26.260808 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:26.260774    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:30.200207 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:30.200167    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:08:30.922109 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:30.922070    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:08:30.922326 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:08:30.922232    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:08:31.396653 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.396556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:31.426679 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.426640    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"]
Sep 18 20:08:31.430305 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.430273    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.432495 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.432471    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 18 20:08:31.432626 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.432555    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 18 20:08:31.440415 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.440389    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"]
Sep 18 20:08:31.485998 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.485967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.486206 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.486037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.486206 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.486102    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8cb0e3f-8d99-4649-a303-5796021d72d4-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.486335 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.486207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.486335 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.486271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgk8g\" (UniqueName: \"kubernetes.io/projected/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-api-access-dgk8g\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.587663 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.587611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.587663 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.587667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8cb0e3f-8d99-4649-a303-5796021d72d4-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.587965 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.587716    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.587965 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.587765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgk8g\" (UniqueName: \"kubernetes.io/projected/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-api-access-dgk8g\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.587965 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.587814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.588081 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:08:31.587987    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 18 20:08:31.588081 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:08:31.588056    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls podName:c8cb0e3f-8d99-4649-a303-5796021d72d4 nodeName:}" failed. No retries permitted until 2026-09-18 20:08:32.088040848 +0000 UTC m=+1819.690404008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-kf5wp" (UID: "c8cb0e3f-8d99-4649-a303-5796021d72d4") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 18 20:08:31.588252 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.588230    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8cb0e3f-8d99-4649-a303-5796021d72d4-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.588508 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.588484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.588548 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.588523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.598172 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.598142    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgk8g\" (UniqueName: \"kubernetes.io/projected/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-api-access-dgk8g\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:31.765671 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:31.765638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:32.092139 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:32.092037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:32.094503 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:32.094467    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-kf5wp\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:32.341431 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:32.341389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:32.616935 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:32.616885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"]
Sep 18 20:08:32.619445 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:08:32.619413    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cb0e3f_8d99_4649_a303_5796021d72d4.slice/crio-e623446a6bc6e2e083b6b79a20a5452b274d58ae34cf62bc184d494b43477191 WatchSource:0}: Error finding container e623446a6bc6e2e083b6b79a20a5452b274d58ae34cf62bc184d494b43477191: Status 404 returned error can't find the container with id e623446a6bc6e2e083b6b79a20a5452b274d58ae34cf62bc184d494b43477191
Sep 18 20:08:33.415581 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:33.415547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" event={"ID":"c8cb0e3f-8d99-4649-a303-5796021d72d4","Type":"ContainerStarted","Data":"3cbb9aee17858d428cced71aba8b951b1942d6be82ed949776adef675e0b0a14"}
Sep 18 20:08:33.415781 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:33.415603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" event={"ID":"c8cb0e3f-8d99-4649-a303-5796021d72d4","Type":"ContainerStarted","Data":"e623446a6bc6e2e083b6b79a20a5452b274d58ae34cf62bc184d494b43477191"}
Sep 18 20:08:37.439628 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:37.439590    2560 generic.go:356] "Generic (PLEG): container finished" podID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerID="3cbb9aee17858d428cced71aba8b951b1942d6be82ed949776adef675e0b0a14" exitCode=0
Sep 18 20:08:37.440053 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:37.439646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" event={"ID":"c8cb0e3f-8d99-4649-a303-5796021d72d4","Type":"ContainerDied","Data":"3cbb9aee17858d428cced71aba8b951b1942d6be82ed949776adef675e0b0a14"}
Sep 18 20:08:37.440911 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:37.440892    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:08:40.199484 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:40.199445    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:08:41.462606 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:41.462572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" event={"ID":"c8cb0e3f-8d99-4649-a303-5796021d72d4","Type":"ContainerStarted","Data":"37c831dd43890b9c6a323985853b1e9af8914dc0cad1f3ba90a7d20dd4626b8a"}
Sep 18 20:08:41.462999 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:41.462633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" event={"ID":"c8cb0e3f-8d99-4649-a303-5796021d72d4","Type":"ContainerStarted","Data":"1c3402171771e022a344ae292112893ba248585c05b66ed721f4e5608baa9099"}
Sep 18 20:08:41.462999 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:41.462827    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:41.482500 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:41.482447    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podStartSLOduration=2.365699577 podStartE2EDuration="10.482426773s" totalImagesPullingTime="3.658623931s" totalInitContainerRuntime="4.458103265s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:08:41.480357847 +0000 UTC m=+1829.082721030" watchObservedRunningTime="2026-09-18 20:08:41.482426773 +0000 UTC m=+1829.084789956"
Sep 18 20:08:42.467380 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:42.467348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:42.468625 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:42.468598    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:08:43.472015 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:43.471973    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:08:43.922521 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:43.922494    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:08:43.922691 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:08:43.922665    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:08:48.476661 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:48.476631    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:48.477216 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:48.477185    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:08:50.199923 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:50.199879    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:08:58.477666 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:58.477636    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:08:58.922970 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:58.922941    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:08:58.923199 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:08:58.923174    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:08:59.923628 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:08:59.923597    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:09:07.241175 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:07.241142    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"]
Sep 18 20:09:07.241587 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:07.241386    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" containerID="cri-o://b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3" gracePeriod=30
Sep 18 20:09:07.241587 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:07.241437    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kube-rbac-proxy" containerID="cri-o://d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b" gracePeriod=30
Sep 18 20:09:07.266018 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:07.265984    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:07.582472 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:07.582377    2560 generic.go:356] "Generic (PLEG): container finished" podID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerID="d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b" exitCode=2
Sep 18 20:09:07.582472 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:07.582446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" event={"ID":"8e4c0fd4-83ab-45d8-aba6-7e371d973f56","Type":"ContainerDied","Data":"d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b"}
Sep 18 20:09:09.922673 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:09.922620    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:09:10.194308 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:10.194211    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 18 20:09:10.922306 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:10.922274    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:09:10.935054 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:09:10.922447    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:09:12.395580 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.395548    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:09:12.410076 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.410033    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:12.447673 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.447582    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-rbac-proxy-sar-config\") pod \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") "
Sep 18 20:09:12.447673 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.447622    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpzg9\" (UniqueName: \"kubernetes.io/projected/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-api-access-qpzg9\") pod \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") "
Sep 18 20:09:12.447931 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.447683    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-proxy-tls\") pod \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") "
Sep 18 20:09:12.447931 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.447750    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-cabundle-cert\") pod \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") "
Sep 18 20:09:12.447931 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.447790    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kserve-provision-location\") pod \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\" (UID: \"8e4c0fd4-83ab-45d8-aba6-7e371d973f56\") "
Sep 18 20:09:12.448171 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.448141    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e4c0fd4-83ab-45d8-aba6-7e371d973f56" (UID: "8e4c0fd4-83ab-45d8-aba6-7e371d973f56"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:12.448284 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.448143    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e4c0fd4-83ab-45d8-aba6-7e371d973f56" (UID: "8e4c0fd4-83ab-45d8-aba6-7e371d973f56"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:12.448284 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.448224    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e4c0fd4-83ab-45d8-aba6-7e371d973f56" (UID: "8e4c0fd4-83ab-45d8-aba6-7e371d973f56"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:12.450068 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.450033    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e4c0fd4-83ab-45d8-aba6-7e371d973f56" (UID: "8e4c0fd4-83ab-45d8-aba6-7e371d973f56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:09:12.450068 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.450046    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-api-access-qpzg9" (OuterVolumeSpecName: "kube-api-access-qpzg9") pod "8e4c0fd4-83ab-45d8-aba6-7e371d973f56" (UID: "8e4c0fd4-83ab-45d8-aba6-7e371d973f56"). InnerVolumeSpecName "kube-api-access-qpzg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:12.481162 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.481125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:12.522532 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522496    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"]
Sep 18 20:09:12.522879 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522865    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container"
Sep 18 20:09:12.522938 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522880    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container"
Sep 18 20:09:12.522938 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522890    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kube-rbac-proxy"
Sep 18 20:09:12.522938 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522895    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kube-rbac-proxy"
Sep 18 20:09:12.522938 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522907    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="storage-initializer"
Sep 18 20:09:12.522938 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522913    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="storage-initializer"
Sep 18 20:09:12.523084 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522966    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kube-rbac-proxy"
Sep 18 20:09:12.523084 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.522976    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerName="kserve-container"
Sep 18 20:09:12.525460 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.525435    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.528017 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.527986    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:09:12.528140 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.528015    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 20:09:12.537146 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.537116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"]
Sep 18 20:09:12.549377 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.549350    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.549377 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.549376    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.549566 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.549386    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.549566 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.549399    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpzg9\" (UniqueName: \"kubernetes.io/projected/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-kube-api-access-qpzg9\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.549566 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.549408    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e4c0fd4-83ab-45d8-aba6-7e371d973f56-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.551893 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.551868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:12.608114 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.608076    2560 generic.go:356] "Generic (PLEG): container finished" podID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" containerID="b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3" exitCode=0
Sep 18 20:09:12.608114 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.608117    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" event={"ID":"8e4c0fd4-83ab-45d8-aba6-7e371d973f56","Type":"ContainerDied","Data":"b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3"}
Sep 18 20:09:12.608312 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.608132    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"
Sep 18 20:09:12.608312 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.608167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z" event={"ID":"8e4c0fd4-83ab-45d8-aba6-7e371d973f56","Type":"ContainerDied","Data":"f07142703ace68f47caa8933fb0912d5b4aba98305bd096f5392704861b65f8b"}
Sep 18 20:09:12.608312 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.608195    2560 scope.go:117] "RemoveContainer" containerID="d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b"
Sep 18 20:09:12.617391 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.617372    2560 scope.go:117] "RemoveContainer" containerID="b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3"
Sep 18 20:09:12.625355 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.625334    2560 scope.go:117] "RemoveContainer" containerID="6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d"
Sep 18 20:09:12.630937 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.630912    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"]
Sep 18 20:09:12.633946 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.633929    2560 scope.go:117] "RemoveContainer" containerID="d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b"
Sep 18 20:09:12.634281 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:09:12.634261    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b\": container with ID starting with d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b not found: ID does not exist" containerID="d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b"
Sep 18 20:09:12.634351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.634289    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b"} err="failed to get container status \"d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b\": rpc error: code = NotFound desc = could not find container \"d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b\": container with ID starting with d793f49d196b38ec1a1b81ce75da8105da70777c6518035907eeab53e7a7043b not found: ID does not exist"
Sep 18 20:09:12.634351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.634311    2560 scope.go:117] "RemoveContainer" containerID="b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3"
Sep 18 20:09:12.634593 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:09:12.634574    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3\": container with ID starting with b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3 not found: ID does not exist" containerID="b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3"
Sep 18 20:09:12.634637 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.634600    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3"} err="failed to get container status \"b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3\": rpc error: code = NotFound desc = could not find container \"b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3\": container with ID starting with b3639b10ec04f69926a1d509d375536ae27f84e24cd262fd04586b2148c0caf3 not found: ID does not exist"
Sep 18 20:09:12.634637 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.634616    2560 scope.go:117] "RemoveContainer" containerID="6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d"
Sep 18 20:09:12.634964 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:09:12.634842    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d\": container with ID starting with 6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d not found: ID does not exist" containerID="6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d"
Sep 18 20:09:12.635032 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.634968    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d"} err="failed to get container status \"6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d\": rpc error: code = NotFound desc = could not find container \"6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d\": container with ID starting with 6f58202ec57b8bc4bf5c43d17067fa9bb95835857aac0631a05aa7168164fa3d not found: ID does not exist"
Sep 18 20:09:12.638318 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.638292    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z"]
Sep 18 20:09:12.649947 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.649918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.650069 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.649951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.650069 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.650000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6050e545-1c62-43b3-93e1-53f808a6f673-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.650152 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.650060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lltf\" (UniqueName: \"kubernetes.io/projected/6050e545-1c62-43b3-93e1-53f808a6f673-kube-api-access-4lltf\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.650152 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.650112    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6050e545-1c62-43b3-93e1-53f808a6f673-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.720046 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.719919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:12.751211 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.751178    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6050e545-1c62-43b3-93e1-53f808a6f673-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.751391 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.751221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lltf\" (UniqueName: \"kubernetes.io/projected/6050e545-1c62-43b3-93e1-53f808a6f673-kube-api-access-4lltf\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.751391 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.751256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6050e545-1c62-43b3-93e1-53f808a6f673-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.751391 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.751283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.751391 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.751299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.751705 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.751682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6050e545-1c62-43b3-93e1-53f808a6f673-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.751944 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.751926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.753504 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.753480    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:09:12.753613 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.753526    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 20:09:12.759155 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.759135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lltf\" (UniqueName: \"kubernetes.io/projected/6050e545-1c62-43b3-93e1-53f808a6f673-kube-api-access-4lltf\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.762388 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.762366    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.763944 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.763924    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6050e545-1c62-43b3-93e1-53f808a6f673-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-88wtv\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.838604 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.838544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:12.928150 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:12.928116    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e4c0fd4-83ab-45d8-aba6-7e371d973f56" path="/var/lib/kubelet/pods/8e4c0fd4-83ab-45d8-aba6-7e371d973f56/volumes"
Sep 18 20:09:13.110432 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.110406    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"]
Sep 18 20:09:13.112286 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:09:13.112254    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6050e545_1c62_43b3_93e1_53f808a6f673.slice/crio-9aca01b10b4da2c15480402211988154b747f798ec375275f457cf337157d8e9 WatchSource:0}: Error finding container 9aca01b10b4da2c15480402211988154b747f798ec375275f457cf337157d8e9: Status 404 returned error can't find the container with id 9aca01b10b4da2c15480402211988154b747f798ec375275f457cf337157d8e9
Sep 18 20:09:13.188500 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.188468    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"]
Sep 18 20:09:13.188780 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.188732    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kserve-container" containerID="cri-o://1c3402171771e022a344ae292112893ba248585c05b66ed721f4e5608baa9099" gracePeriod=30
Sep 18 20:09:13.189004 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.188785    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy" containerID="cri-o://37c831dd43890b9c6a323985853b1e9af8914dc0cad1f3ba90a7d20dd4626b8a" gracePeriod=30
Sep 18 20:09:13.210758 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.210731    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:13.473065 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.473025    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 18 20:09:13.615711 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.615671    2560 generic.go:356] "Generic (PLEG): container finished" podID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerID="37c831dd43890b9c6a323985853b1e9af8914dc0cad1f3ba90a7d20dd4626b8a" exitCode=2
Sep 18 20:09:13.615926 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.615736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" event={"ID":"c8cb0e3f-8d99-4649-a303-5796021d72d4","Type":"ContainerDied","Data":"37c831dd43890b9c6a323985853b1e9af8914dc0cad1f3ba90a7d20dd4626b8a"}
Sep 18 20:09:13.617458 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.617417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" event={"ID":"6050e545-1c62-43b3-93e1-53f808a6f673","Type":"ContainerStarted","Data":"8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a"}
Sep 18 20:09:13.617615 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:13.617481    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" event={"ID":"6050e545-1c62-43b3-93e1-53f808a6f673","Type":"ContainerStarted","Data":"9aca01b10b4da2c15480402211988154b747f798ec375275f457cf337157d8e9"}
Sep 18 20:09:17.636148 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:17.636110    2560 generic.go:356] "Generic (PLEG): container finished" podID="6050e545-1c62-43b3-93e1-53f808a6f673" containerID="8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a" exitCode=0
Sep 18 20:09:17.636545 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:17.636182    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" event={"ID":"6050e545-1c62-43b3-93e1-53f808a6f673","Type":"ContainerDied","Data":"8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a"}
Sep 18 20:09:18.345512 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:18.345478    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:18.472199 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:18.472154    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 18 20:09:18.584324 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:18.584259    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:18.643463 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:18.643372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" event={"ID":"6050e545-1c62-43b3-93e1-53f808a6f673","Type":"ContainerStarted","Data":"31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67"}
Sep 18 20:09:18.643463 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:18.643440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" event={"ID":"6050e545-1c62-43b3-93e1-53f808a6f673","Type":"ContainerStarted","Data":"7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38"}
Sep 18 20:09:18.643883 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:18.643642    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:18.660761 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:18.660710    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podStartSLOduration=2.831697853 podStartE2EDuration="6.660695991s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.828998138s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:18.660044276 +0000 UTC m=+1866.262407459" watchObservedRunningTime="2026-09-18 20:09:18.660695991 +0000 UTC m=+1866.263059173"
Sep 18 20:09:19.647671 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:19.647639    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:19.649240 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:19.649212    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:09:20.652931 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:20.652893    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:09:23.472918 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:23.472840    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 18 20:09:23.473489 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:23.473012    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:09:23.922372 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:23.922335    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:09:23.922538 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:09:23.922478    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:09:25.657957 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:25.657926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:09:25.658466 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:25.658439    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:09:28.472515 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:28.472474    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 18 20:09:33.473004 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:33.472957    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 18 20:09:35.658427 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:35.658345    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:09:38.472919 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:38.472826    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 18 20:09:38.922494 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:38.922464    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:09:38.922693 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:09:38.922635    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:09:43.472796 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:43.472693    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 18 20:09:43.767515 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:43.767477    2560 generic.go:356] "Generic (PLEG): container finished" podID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerID="1c3402171771e022a344ae292112893ba248585c05b66ed721f4e5608baa9099" exitCode=137
Sep 18 20:09:43.767719 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:43.767547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" event={"ID":"c8cb0e3f-8d99-4649-a303-5796021d72d4","Type":"ContainerDied","Data":"1c3402171771e022a344ae292112893ba248585c05b66ed721f4e5608baa9099"}
Sep 18 20:09:43.837598 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:43.837570    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:09:44.018600 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.018504    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8cb0e3f-8d99-4649-a303-5796021d72d4-kserve-provision-location\") pod \"c8cb0e3f-8d99-4649-a303-5796021d72d4\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") "
Sep 18 20:09:44.018600 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.018573    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-rbac-proxy-sar-config\") pod \"c8cb0e3f-8d99-4649-a303-5796021d72d4\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") "
Sep 18 20:09:44.018600 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.018601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgk8g\" (UniqueName: \"kubernetes.io/projected/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-api-access-dgk8g\") pod \"c8cb0e3f-8d99-4649-a303-5796021d72d4\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") "
Sep 18 20:09:44.018908 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.018620    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls\") pod \"c8cb0e3f-8d99-4649-a303-5796021d72d4\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") "
Sep 18 20:09:44.018908 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.018646    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-cabundle-cert\") pod \"c8cb0e3f-8d99-4649-a303-5796021d72d4\" (UID: \"c8cb0e3f-8d99-4649-a303-5796021d72d4\") "
Sep 18 20:09:44.019050 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.019024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8cb0e3f-8d99-4649-a303-5796021d72d4" (UID: "c8cb0e3f-8d99-4649-a303-5796021d72d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:44.019151 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.019127    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8cb0e3f-8d99-4649-a303-5796021d72d4" (UID: "c8cb0e3f-8d99-4649-a303-5796021d72d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:44.020763 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.020737    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8cb0e3f-8d99-4649-a303-5796021d72d4" (UID: "c8cb0e3f-8d99-4649-a303-5796021d72d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:09:44.020879 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.020840    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-api-access-dgk8g" (OuterVolumeSpecName: "kube-api-access-dgk8g") pod "c8cb0e3f-8d99-4649-a303-5796021d72d4" (UID: "c8cb0e3f-8d99-4649-a303-5796021d72d4"). InnerVolumeSpecName "kube-api-access-dgk8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:44.029245 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.029215    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cb0e3f-8d99-4649-a303-5796021d72d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8cb0e3f-8d99-4649-a303-5796021d72d4" (UID: "c8cb0e3f-8d99-4649-a303-5796021d72d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:44.119990 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.119945    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8cb0e3f-8d99-4649-a303-5796021d72d4-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:44.119990 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.119984    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:44.119990 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.119994    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgk8g\" (UniqueName: \"kubernetes.io/projected/c8cb0e3f-8d99-4649-a303-5796021d72d4-kube-api-access-dgk8g\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:44.120247 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.120003    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8cb0e3f-8d99-4649-a303-5796021d72d4-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:44.120247 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.120015    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8cb0e3f-8d99-4649-a303-5796021d72d4-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:09:44.774411 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.774369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp" event={"ID":"c8cb0e3f-8d99-4649-a303-5796021d72d4","Type":"ContainerDied","Data":"e623446a6bc6e2e083b6b79a20a5452b274d58ae34cf62bc184d494b43477191"}
Sep 18 20:09:44.774881 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.774390    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"
Sep 18 20:09:44.774881 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.774477    2560 scope.go:117] "RemoveContainer" containerID="37c831dd43890b9c6a323985853b1e9af8914dc0cad1f3ba90a7d20dd4626b8a"
Sep 18 20:09:44.783793 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.783771    2560 scope.go:117] "RemoveContainer" containerID="1c3402171771e022a344ae292112893ba248585c05b66ed721f4e5608baa9099"
Sep 18 20:09:44.791367 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.791347    2560 scope.go:117] "RemoveContainer" containerID="3cbb9aee17858d428cced71aba8b951b1942d6be82ed949776adef675e0b0a14"
Sep 18 20:09:44.795932 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.795901    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"]
Sep 18 20:09:44.799559 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.799536    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp"]
Sep 18 20:09:44.927700 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:44.927657    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" path="/var/lib/kubelet/pods/c8cb0e3f-8d99-4649-a303-5796021d72d4/volumes"
Sep 18 20:09:45.658461 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:45.658424    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:09:52.924709 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:52.924679    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:09:53.819159 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:53.819076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"}
Sep 18 20:09:53.819316 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:53.819282    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:09:55.538132 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:55.538097    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:55.640125 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:55.640088    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:55.658643 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:09:55.658599    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:10:00.169610 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:00.169575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:05.374574 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.374537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:05.395498 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395466    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"]
Sep 18 20:10:05.395832 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395817    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="storage-initializer"
Sep 18 20:10:05.395896 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395836    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="storage-initializer"
Sep 18 20:10:05.395896 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395857    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy"
Sep 18 20:10:05.395896 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395863    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy"
Sep 18 20:10:05.395896 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395876    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kserve-container"
Sep 18 20:10:05.395896 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395881    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kserve-container"
Sep 18 20:10:05.396053 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395938    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kserve-container"
Sep 18 20:10:05.396053 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.395948    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8cb0e3f-8d99-4649-a303-5796021d72d4" containerName="kube-rbac-proxy"
Sep 18 20:10:05.399307 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.399287    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.401790 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.401772    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 18 20:10:05.401916 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.401770    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 18 20:10:05.410882 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.410834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"]
Sep 18 20:10:05.495169 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.495118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.495169 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.495173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.495391 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.495224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.495391 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.495272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbdn\" (UniqueName: \"kubernetes.io/projected/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-api-access-6mbdn\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.495391 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.495294    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.596481 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.596445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.596481 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.596482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.596690 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.596597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbdn\" (UniqueName: \"kubernetes.io/projected/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-api-access-6mbdn\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.596690 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.596630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.596772 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.596756    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.596861 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:10:05.596823    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 18 20:10:05.596920 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.596835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.596920 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:10:05.596916    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls podName:9fbc921e-91c8-4ef3-a5d3-22150040cb25 nodeName:}" failed. No retries permitted until 2026-09-18 20:10:06.096897419 +0000 UTC m=+1913.699260579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls") pod "isvc-triton-predictor-5d7db58676-zz2pj" (UID: "9fbc921e-91c8-4ef3-a5d3-22150040cb25") : secret "isvc-triton-predictor-serving-cert" not found
Sep 18 20:10:05.597225 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.597208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.597350 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.597333    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.623995 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.623970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbdn\" (UniqueName: \"kubernetes.io/projected/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-api-access-6mbdn\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:05.659410 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.659364    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:10:05.727422 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:05.727380    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:06.101014 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:06.100920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:06.103398 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:06.103372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-zz2pj\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:06.309705 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:06.309664    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:10:06.584326 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:06.584269    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"]
Sep 18 20:10:06.586531 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:10:06.586495    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbc921e_91c8_4ef3_a5d3_22150040cb25.slice/crio-1a6ec4bfb06e9dfdbbc8938b6eeff2d5e648d4bde115ef7631049decc9c9e1c8 WatchSource:0}: Error finding container 1a6ec4bfb06e9dfdbbc8938b6eeff2d5e648d4bde115ef7631049decc9c9e1c8: Status 404 returned error can't find the container with id 1a6ec4bfb06e9dfdbbc8938b6eeff2d5e648d4bde115ef7631049decc9c9e1c8
Sep 18 20:10:06.885864 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:06.885750    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" event={"ID":"9fbc921e-91c8-4ef3-a5d3-22150040cb25","Type":"ContainerStarted","Data":"a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3"}
Sep 18 20:10:06.885864 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:06.885808    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" event={"ID":"9fbc921e-91c8-4ef3-a5d3-22150040cb25","Type":"ContainerStarted","Data":"1a6ec4bfb06e9dfdbbc8938b6eeff2d5e648d4bde115ef7631049decc9c9e1c8"}
Sep 18 20:10:10.904692 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:10.904654    2560 generic.go:356] "Generic (PLEG): container finished" podID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerID="a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3" exitCode=0
Sep 18 20:10:10.905138 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:10.904730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" event={"ID":"9fbc921e-91c8-4ef3-a5d3-22150040cb25","Type":"ContainerDied","Data":"a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3"}
Sep 18 20:10:15.658624 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:15.658570    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:10:24.827097 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:24.826982    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:10:25.658479 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:25.658416    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:10:35.658838 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:35.658430    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:10:44.929916 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:44.929822    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:10:45.895843 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:45.895802    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"]
Sep 18 20:10:46.141731 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:46.141282    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container" containerID="cri-o://7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38" gracePeriod=30
Sep 18 20:10:46.141731 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:46.141453    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kube-rbac-proxy" containerID="cri-o://31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67" gracePeriod=30
Sep 18 20:10:47.150151 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:47.150098    2560 generic.go:356] "Generic (PLEG): container finished" podID="6050e545-1c62-43b3-93e1-53f808a6f673" containerID="31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67" exitCode=2
Sep 18 20:10:47.150660 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:47.150169    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" event={"ID":"6050e545-1c62-43b3-93e1-53f808a6f673","Type":"ContainerDied","Data":"31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67"}
Sep 18 20:10:50.653870 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:50.653809    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 18 20:10:52.155506 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.155470    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:10:52.184194 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.184149    2560 generic.go:356] "Generic (PLEG): container finished" podID="6050e545-1c62-43b3-93e1-53f808a6f673" containerID="7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38" exitCode=0
Sep 18 20:10:52.184386 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.184236    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"
Sep 18 20:10:52.184386 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.184309    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" event={"ID":"6050e545-1c62-43b3-93e1-53f808a6f673","Type":"ContainerDied","Data":"7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38"}
Sep 18 20:10:52.184386 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.184372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv" event={"ID":"6050e545-1c62-43b3-93e1-53f808a6f673","Type":"ContainerDied","Data":"9aca01b10b4da2c15480402211988154b747f798ec375275f457cf337157d8e9"}
Sep 18 20:10:52.184564 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.184412    2560 scope.go:117] "RemoveContainer" containerID="31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67"
Sep 18 20:10:52.197403 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.197374    2560 scope.go:117] "RemoveContainer" containerID="7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38"
Sep 18 20:10:52.208528 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.208504    2560 scope.go:117] "RemoveContainer" containerID="8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a"
Sep 18 20:10:52.224052 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.224021    2560 scope.go:117] "RemoveContainer" containerID="31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67"
Sep 18 20:10:52.225407 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:10:52.225338    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67\": container with ID starting with 31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67 not found: ID does not exist" containerID="31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67"
Sep 18 20:10:52.225407 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.225381    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67"} err="failed to get container status \"31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67\": rpc error: code = NotFound desc = could not find container \"31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67\": container with ID starting with 31c953325d2019d747f5b2992aae25553c51a46330427b7f3d95f0c5a2f1df67 not found: ID does not exist"
Sep 18 20:10:52.225407 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.225409    2560 scope.go:117] "RemoveContainer" containerID="7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38"
Sep 18 20:10:52.225971 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:10:52.225912    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38\": container with ID starting with 7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38 not found: ID does not exist" containerID="7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38"
Sep 18 20:10:52.225971 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.225947    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38"} err="failed to get container status \"7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38\": rpc error: code = NotFound desc = could not find container \"7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38\": container with ID starting with 7e5895434f696bfcffb085a5affa0262d9e877fa2a3201ec12e15a679f301b38 not found: ID does not exist"
Sep 18 20:10:52.225971 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.225972    2560 scope.go:117] "RemoveContainer" containerID="8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a"
Sep 18 20:10:52.226835 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:10:52.226760    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a\": container with ID starting with 8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a not found: ID does not exist" containerID="8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a"
Sep 18 20:10:52.226835 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.226808    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a"} err="failed to get container status \"8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a\": rpc error: code = NotFound desc = could not find container \"8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a\": container with ID starting with 8505751c2e74e8c0718eb87d381e5cdc51b27acb029359d0f5a63defca963c5a not found: ID does not exist"
Sep 18 20:10:52.259333 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.259300    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-cabundle-cert\") pod \"6050e545-1c62-43b3-93e1-53f808a6f673\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") "
Sep 18 20:10:52.259590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.259379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6050e545-1c62-43b3-93e1-53f808a6f673-proxy-tls\") pod \"6050e545-1c62-43b3-93e1-53f808a6f673\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") "
Sep 18 20:10:52.259590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.259414    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6050e545-1c62-43b3-93e1-53f808a6f673-kserve-provision-location\") pod \"6050e545-1c62-43b3-93e1-53f808a6f673\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") "
Sep 18 20:10:52.259590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.259439    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lltf\" (UniqueName: \"kubernetes.io/projected/6050e545-1c62-43b3-93e1-53f808a6f673-kube-api-access-4lltf\") pod \"6050e545-1c62-43b3-93e1-53f808a6f673\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") "
Sep 18 20:10:52.259590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.259480    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-kube-rbac-proxy-sar-config\") pod \"6050e545-1c62-43b3-93e1-53f808a6f673\" (UID: \"6050e545-1c62-43b3-93e1-53f808a6f673\") "
Sep 18 20:10:52.260321 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.260116    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6050e545-1c62-43b3-93e1-53f808a6f673" (UID: "6050e545-1c62-43b3-93e1-53f808a6f673"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:52.260321 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.260161    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6050e545-1c62-43b3-93e1-53f808a6f673-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6050e545-1c62-43b3-93e1-53f808a6f673" (UID: "6050e545-1c62-43b3-93e1-53f808a6f673"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:52.260532 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.260398    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6050e545-1c62-43b3-93e1-53f808a6f673" (UID: "6050e545-1c62-43b3-93e1-53f808a6f673"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:52.262908 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.262617    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6050e545-1c62-43b3-93e1-53f808a6f673-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6050e545-1c62-43b3-93e1-53f808a6f673" (UID: "6050e545-1c62-43b3-93e1-53f808a6f673"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:52.263408 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.263374    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6050e545-1c62-43b3-93e1-53f808a6f673-kube-api-access-4lltf" (OuterVolumeSpecName: "kube-api-access-4lltf") pod "6050e545-1c62-43b3-93e1-53f808a6f673" (UID: "6050e545-1c62-43b3-93e1-53f808a6f673"). InnerVolumeSpecName "kube-api-access-4lltf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:52.361085 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.361045    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lltf\" (UniqueName: \"kubernetes.io/projected/6050e545-1c62-43b3-93e1-53f808a6f673-kube-api-access-4lltf\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:10:52.361085 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.361084    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:10:52.361351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.361100    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6050e545-1c62-43b3-93e1-53f808a6f673-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:10:52.361351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.361114    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6050e545-1c62-43b3-93e1-53f808a6f673-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:10:52.361351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.361127    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6050e545-1c62-43b3-93e1-53f808a6f673-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:10:52.465746 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.465548    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:52.514376 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.514250    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"]
Sep 18 20:10:52.516739 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.516717    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv"]
Sep 18 20:10:52.536203 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536160    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"]
Sep 18 20:10:52.536634 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536615    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="storage-initializer"
Sep 18 20:10:52.536634 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536635    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="storage-initializer"
Sep 18 20:10:52.536810 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536666    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container"
Sep 18 20:10:52.536810 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536674    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container"
Sep 18 20:10:52.536810 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536696    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kube-rbac-proxy"
Sep 18 20:10:52.536810 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536704    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kube-rbac-proxy"
Sep 18 20:10:52.536810 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536789    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kube-rbac-proxy"
Sep 18 20:10:52.537424 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.536813    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" containerName="kserve-container"
Sep 18 20:10:52.553145 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.553095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"]
Sep 18 20:10:52.553345 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.553235    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.555665 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.555581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:10:52.556278 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.555830    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 20:10:52.664647 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.664350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.664647 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.664414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.664647 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.664477    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.664647 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.664502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.664647 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.664586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtc6\" (UniqueName: \"kubernetes.io/projected/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-api-access-2vtc6\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.765365 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.765243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.765365 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.765299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.765735 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.765385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtc6\" (UniqueName: \"kubernetes.io/projected/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-api-access-2vtc6\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.765735 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.765417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.765735 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.765451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.766484 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.766168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.766484 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.766408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.766864 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.766811    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.768495 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.768473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.774561 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.774507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtc6\" (UniqueName: \"kubernetes.io/projected/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-api-access-2vtc6\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l6xcg\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.868884 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.868805    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:10:52.932960 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:52.932445    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6050e545-1c62-43b3-93e1-53f808a6f673" path="/var/lib/kubelet/pods/6050e545-1c62-43b3-93e1-53f808a6f673/volumes"
Sep 18 20:10:53.189106 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:53.189078    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"]
Sep 18 20:10:53.192245 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:10:53.192194    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc70fe2f_f0e3_4bc8_b015_a01622a5592a.slice/crio-b1a29dd65d732174c4a4b71cc71a28e5bf2e145e06b56b80529c96f676132f29 WatchSource:0}: Error finding container b1a29dd65d732174c4a4b71cc71a28e5bf2e145e06b56b80529c96f676132f29: Status 404 returned error can't find the container with id b1a29dd65d732174c4a4b71cc71a28e5bf2e145e06b56b80529c96f676132f29
Sep 18 20:10:54.200649 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:54.200524    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" event={"ID":"dc70fe2f-f0e3-4bc8-b015-a01622a5592a","Type":"ContainerStarted","Data":"4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40"}
Sep 18 20:10:54.200649 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:10:54.200599    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" event={"ID":"dc70fe2f-f0e3-4bc8-b015-a01622a5592a","Type":"ContainerStarted","Data":"b1a29dd65d732174c4a4b71cc71a28e5bf2e145e06b56b80529c96f676132f29"}
Sep 18 20:11:24.357156 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:11:24.357120    2560 generic.go:356] "Generic (PLEG): container finished" podID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerID="4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40" exitCode=0
Sep 18 20:11:24.357623 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:11:24.357202    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" event={"ID":"dc70fe2f-f0e3-4bc8-b015-a01622a5592a","Type":"ContainerDied","Data":"4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40"}
Sep 18 20:11:35.422189 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:11:35.422136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" event={"ID":"dc70fe2f-f0e3-4bc8-b015-a01622a5592a","Type":"ContainerStarted","Data":"df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80"}
Sep 18 20:12:05.578446 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:05.578401    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" event={"ID":"dc70fe2f-f0e3-4bc8-b015-a01622a5592a","Type":"ContainerStarted","Data":"011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f"}
Sep 18 20:12:05.578920 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:05.578603    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:12:05.598445 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:05.598375    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podStartSLOduration=69.354284946 podStartE2EDuration="1m13.598351646s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.2440667s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:05.59550583 +0000 UTC m=+2033.197869014" watchObservedRunningTime="2026-09-18 20:12:05.598351646 +0000 UTC m=+2033.200714828"
Sep 18 20:12:06.586073 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:06.586036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" event={"ID":"9fbc921e-91c8-4ef3-a5d3-22150040cb25","Type":"ContainerStarted","Data":"c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c"}
Sep 18 20:12:06.586508 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:06.586107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" event={"ID":"9fbc921e-91c8-4ef3-a5d3-22150040cb25","Type":"ContainerStarted","Data":"b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326"}
Sep 18 20:12:06.586508 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:06.586475    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:12:06.587611 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:06.587584    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:12:06.605887 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:06.605821    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" podStartSLOduration=3.111003542 podStartE2EDuration="2m1.605804095s" totalImagesPullingTime="1m54.851087251s" totalInitContainerRuntime="3.643713302s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:06.603678324 +0000 UTC m=+2034.206041504" watchObservedRunningTime="2026-09-18 20:12:06.605804095 +0000 UTC m=+2034.208167277"
Sep 18 20:12:07.591602 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:07.591566    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:12:07.592047 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:07.591812    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:12:07.592047 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:07.591826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:12:07.593101 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:07.593076    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:12:08.596369 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:08.596331    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:12:12.596159 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:12.596077    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:12:12.596614 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:12.596584    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:12:12.617316 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:12.617198    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517" exitCode=1
Sep 18 20:12:12.617316 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:12.617271    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"}
Sep 18 20:12:12.617797 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:12.617352    2560 scope.go:117] "RemoveContainer" containerID="481419ac67ba0e6d59b134b0d64d8710dd29b9ffed3d47e959cac6c388cd526b"
Sep 18 20:12:12.617797 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:12.617737    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:12:12.617986 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:12.617964    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:12:13.600559 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:13.600526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:12:13.601538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:13.601520    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:12:14.824778 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:14.824737    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:12:14.825243 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:14.825099    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:12:14.825335 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:14.825312    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:12:17.596605 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:17.596573    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"]
Sep 18 20:12:17.597008 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:17.596813    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kserve-container" containerID="cri-o://b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326" gracePeriod=30
Sep 18 20:12:17.597008 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:17.596869    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kube-rbac-proxy" containerID="cri-o://c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c" gracePeriod=30
Sep 18 20:12:17.612364 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:17.612339    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:18.596779 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:18.596736    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 18 20:12:18.647647 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:18.647612    2560 generic.go:356] "Generic (PLEG): container finished" podID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerID="c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c" exitCode=2
Sep 18 20:12:18.647811 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:18.647684    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" event={"ID":"9fbc921e-91c8-4ef3-a5d3-22150040cb25","Type":"ContainerDied","Data":"c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c"}
Sep 18 20:12:20.088010 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.087972    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:12:20.221599 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.221562    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mbdn\" (UniqueName: \"kubernetes.io/projected/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-api-access-6mbdn\") pod \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") "
Sep 18 20:12:20.221599 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.221601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls\") pod \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") "
Sep 18 20:12:20.221832 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.221669    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-rbac-proxy-sar-config\") pod \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") "
Sep 18 20:12:20.221832 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.221741    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kserve-provision-location\") pod \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") "
Sep 18 20:12:20.221832 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.221763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-cabundle-cert\") pod \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\" (UID: \"9fbc921e-91c8-4ef3-a5d3-22150040cb25\") "
Sep 18 20:12:20.222162 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.222132    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fbc921e-91c8-4ef3-a5d3-22150040cb25" (UID: "9fbc921e-91c8-4ef3-a5d3-22150040cb25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:20.222274 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.222181    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9fbc921e-91c8-4ef3-a5d3-22150040cb25" (UID: "9fbc921e-91c8-4ef3-a5d3-22150040cb25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:20.222274 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.222224    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fbc921e-91c8-4ef3-a5d3-22150040cb25" (UID: "9fbc921e-91c8-4ef3-a5d3-22150040cb25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:20.223755 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.223731    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-api-access-6mbdn" (OuterVolumeSpecName: "kube-api-access-6mbdn") pod "9fbc921e-91c8-4ef3-a5d3-22150040cb25" (UID: "9fbc921e-91c8-4ef3-a5d3-22150040cb25"). InnerVolumeSpecName "kube-api-access-6mbdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:20.223874 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.223794    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fbc921e-91c8-4ef3-a5d3-22150040cb25" (UID: "9fbc921e-91c8-4ef3-a5d3-22150040cb25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:20.322646 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.322607    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:12:20.322646 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.322641    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:12:20.322646 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.322653    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mbdn\" (UniqueName: \"kubernetes.io/projected/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-api-access-6mbdn\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:12:20.322930 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.322664    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fbc921e-91c8-4ef3-a5d3-22150040cb25-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:12:20.322930 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.322675    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fbc921e-91c8-4ef3-a5d3-22150040cb25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:12:20.659482 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.659444    2560 generic.go:356] "Generic (PLEG): container finished" podID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerID="b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326" exitCode=0
Sep 18 20:12:20.659656 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.659493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" event={"ID":"9fbc921e-91c8-4ef3-a5d3-22150040cb25","Type":"ContainerDied","Data":"b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326"}
Sep 18 20:12:20.659656 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.659553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj" event={"ID":"9fbc921e-91c8-4ef3-a5d3-22150040cb25","Type":"ContainerDied","Data":"1a6ec4bfb06e9dfdbbc8938b6eeff2d5e648d4bde115ef7631049decc9c9e1c8"}
Sep 18 20:12:20.659656 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.659509    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"
Sep 18 20:12:20.659656 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.659582    2560 scope.go:117] "RemoveContainer" containerID="c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c"
Sep 18 20:12:20.668351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.668335    2560 scope.go:117] "RemoveContainer" containerID="b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326"
Sep 18 20:12:20.676144 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.676126    2560 scope.go:117] "RemoveContainer" containerID="a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3"
Sep 18 20:12:20.681210 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.681187    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"]
Sep 18 20:12:20.684439 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.684414    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj"]
Sep 18 20:12:20.684891 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.684873    2560 scope.go:117] "RemoveContainer" containerID="c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c"
Sep 18 20:12:20.685210 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:20.685186    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c\": container with ID starting with c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c not found: ID does not exist" containerID="c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c"
Sep 18 20:12:20.685271 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.685225    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c"} err="failed to get container status \"c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c\": rpc error: code = NotFound desc = could not find container \"c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c\": container with ID starting with c0d9e9440a90467571ef115d0725a72d447f2635de40eec8ddad5de11aa9010c not found: ID does not exist"
Sep 18 20:12:20.685271 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.685254    2560 scope.go:117] "RemoveContainer" containerID="b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326"
Sep 18 20:12:20.685529 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:20.685505    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326\": container with ID starting with b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326 not found: ID does not exist" containerID="b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326"
Sep 18 20:12:20.685591 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.685538    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326"} err="failed to get container status \"b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326\": rpc error: code = NotFound desc = could not find container \"b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326\": container with ID starting with b42fa4a95de12461c3364894ddad0b1b3cd074656b39dea14b6dd679fcc75326 not found: ID does not exist"
Sep 18 20:12:20.685591 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.685570    2560 scope.go:117] "RemoveContainer" containerID="a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3"
Sep 18 20:12:20.685895 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:20.685871    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3\": container with ID starting with a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3 not found: ID does not exist" containerID="a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3"
Sep 18 20:12:20.685946 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.685905    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3"} err="failed to get container status \"a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3\": rpc error: code = NotFound desc = could not find container \"a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3\": container with ID starting with a21d42939d0e63790a6909463392e026370fe4abb0ad19099c30f76ff24ccff3 not found: ID does not exist"
Sep 18 20:12:20.928180 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:20.928099    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" path="/var/lib/kubelet/pods/9fbc921e-91c8-4ef3-a5d3-22150040cb25/volumes"
Sep 18 20:12:22.046742 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:22.046688    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:12:22.047194 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:22.047120    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:12:22.047294 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:22.047275    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:12:22.596726 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:22.596684    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:12:22.740193 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:22.740154    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:22.978609 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:22.978572    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:32.596891 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:32.596832    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:12:32.928166 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:32.928135    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:12:32.928352 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:32.928329    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:12:42.597500 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:42.597412    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:12:46.922223 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:46.922191    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:12:46.922608 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:46.922390    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:12:52.597266 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:52.597220    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:12:57.922785 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:12:57.922747    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:12:57.923200 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:12:57.922985    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:13:02.597790 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:02.597759    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:13:11.922578 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:11.922543    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:13:11.923006 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:13:11.922700    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:13:12.604619 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:12.604580    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"]
Sep 18 20:13:12.604971 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:12.604916    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container" containerID="cri-o://df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80" gracePeriod=30
Sep 18 20:13:12.605055 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:12.604926    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kube-rbac-proxy" containerID="cri-o://011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f" gracePeriod=30
Sep 18 20:13:12.636286 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:12.636256    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:12.892588 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:12.892500    2560 generic.go:356] "Generic (PLEG): container finished" podID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerID="011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f" exitCode=2
Sep 18 20:13:12.892588 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:12.892568    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" event={"ID":"dc70fe2f-f0e3-4bc8-b015-a01622a5592a","Type":"ContainerDied","Data":"011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f"}
Sep 18 20:13:17.352213 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.352187    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:13:17.516964 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.516841    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vtc6\" (UniqueName: \"kubernetes.io/projected/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-api-access-2vtc6\") pod \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") "
Sep 18 20:13:17.516964 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.516912    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kserve-provision-location\") pod \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") "
Sep 18 20:13:17.516964 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.516944    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-proxy-tls\") pod \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") "
Sep 18 20:13:17.517250 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.516979    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-cabundle-cert\") pod \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") "
Sep 18 20:13:17.517250 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.517014    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-rbac-proxy-sar-config\") pod \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\" (UID: \"dc70fe2f-f0e3-4bc8-b015-a01622a5592a\") "
Sep 18 20:13:17.517364 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.517316    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc70fe2f-f0e3-4bc8-b015-a01622a5592a" (UID: "dc70fe2f-f0e3-4bc8-b015-a01622a5592a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:17.517425 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.517372    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc70fe2f-f0e3-4bc8-b015-a01622a5592a" (UID: "dc70fe2f-f0e3-4bc8-b015-a01622a5592a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:17.517500 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.517469    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc70fe2f-f0e3-4bc8-b015-a01622a5592a" (UID: "dc70fe2f-f0e3-4bc8-b015-a01622a5592a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:17.519162 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.519127    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc70fe2f-f0e3-4bc8-b015-a01622a5592a" (UID: "dc70fe2f-f0e3-4bc8-b015-a01622a5592a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:17.519268 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.519199    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-api-access-2vtc6" (OuterVolumeSpecName: "kube-api-access-2vtc6") pod "dc70fe2f-f0e3-4bc8-b015-a01622a5592a" (UID: "dc70fe2f-f0e3-4bc8-b015-a01622a5592a"). InnerVolumeSpecName "kube-api-access-2vtc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:17.618231 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.618185    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtc6\" (UniqueName: \"kubernetes.io/projected/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-api-access-2vtc6\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:13:17.618231 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.618221    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:13:17.618231 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.618231    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:13:17.618231 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.618243    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:13:17.618507 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.618251    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc70fe2f-f0e3-4bc8-b015-a01622a5592a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:13:17.801004 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.800904    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:17.831492 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831453    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"]
Sep 18 20:13:17.831880 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831864    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831883    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831897    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="storage-initializer"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831903    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="storage-initializer"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831909    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kserve-container"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831915    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kserve-container"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831929    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="storage-initializer"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831935    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="storage-initializer"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831942    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kube-rbac-proxy"
Sep 18 20:13:17.831945 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831947    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kube-rbac-proxy"
Sep 18 20:13:17.832217 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831953    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kube-rbac-proxy"
Sep 18 20:13:17.832217 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.831959    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kube-rbac-proxy"
Sep 18 20:13:17.832217 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.832008    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kserve-container"
Sep 18 20:13:17.832217 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.832018    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kube-rbac-proxy"
Sep 18 20:13:17.832217 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.832024    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerName="kserve-container"
Sep 18 20:13:17.832217 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.832029    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbc921e-91c8-4ef3-a5d3-22150040cb25" containerName="kube-rbac-proxy"
Sep 18 20:13:17.834241 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.834217    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:17.836635 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.836613    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 18 20:13:17.836635 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.836622    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 20:13:17.845679 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.845652    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"]
Sep 18 20:13:17.872660 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.872622    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"]
Sep 18 20:13:17.873355 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.873326    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:17.875222 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.875205    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:17.877392 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.877374    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 20:13:17.887688 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.887661    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"]
Sep 18 20:13:17.916099 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.916061    2560 generic.go:356] "Generic (PLEG): container finished" podID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" containerID="df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80" exitCode=0
Sep 18 20:13:17.916287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.916136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" event={"ID":"dc70fe2f-f0e3-4bc8-b015-a01622a5592a","Type":"ContainerDied","Data":"df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80"}
Sep 18 20:13:17.916287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.916147    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"
Sep 18 20:13:17.916287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.916194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg" event={"ID":"dc70fe2f-f0e3-4bc8-b015-a01622a5592a","Type":"ContainerDied","Data":"b1a29dd65d732174c4a4b71cc71a28e5bf2e145e06b56b80529c96f676132f29"}
Sep 18 20:13:17.916287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.916227    2560 scope.go:117] "RemoveContainer" containerID="011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f"
Sep 18 20:13:17.920390 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.920366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c03c40-f166-487e-9d3a-b5b351ada4d3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:17.920532 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.920395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:17.920532 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.920428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c03c40-f166-487e-9d3a-b5b351ada4d3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:17.920532 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.920518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:17.920715 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.920567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9kv\" (UniqueName: \"kubernetes.io/projected/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-api-access-8l9kv\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:17.925009 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.924986    2560 scope.go:117] "RemoveContainer" containerID="df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80"
Sep 18 20:13:17.932670 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.932648    2560 scope.go:117] "RemoveContainer" containerID="4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40"
Sep 18 20:13:17.940919 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.940901    2560 scope.go:117] "RemoveContainer" containerID="011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f"
Sep 18 20:13:17.941332 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:13:17.941304    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f\": container with ID starting with 011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f not found: ID does not exist" containerID="011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f"
Sep 18 20:13:17.941424 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.941340    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f"} err="failed to get container status \"011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f\": rpc error: code = NotFound desc = could not find container \"011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f\": container with ID starting with 011c4ecd87d473e37a1f0b08ec1c5e26ceea0421ecd00785de3d7dddd252e98f not found: ID does not exist"
Sep 18 20:13:17.941424 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.941381    2560 scope.go:117] "RemoveContainer" containerID="df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80"
Sep 18 20:13:17.941689 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:13:17.941670    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80\": container with ID starting with df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80 not found: ID does not exist" containerID="df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80"
Sep 18 20:13:17.941741 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.941696    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80"} err="failed to get container status \"df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80\": rpc error: code = NotFound desc = could not find container \"df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80\": container with ID starting with df416037b53200e8bd538e1928d9d04b9b045bb1a35c4797683b92c7e22aaa80 not found: ID does not exist"
Sep 18 20:13:17.941741 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.941719    2560 scope.go:117] "RemoveContainer" containerID="4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40"
Sep 18 20:13:17.942188 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:13:17.942113    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40\": container with ID starting with 4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40 not found: ID does not exist" containerID="4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40"
Sep 18 20:13:17.942188 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.942169    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40"} err="failed to get container status \"4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40\": rpc error: code = NotFound desc = could not find container \"4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40\": container with ID starting with 4c208a95f9420e5907f3d65529382fdd86a2a61bc7af0245771ba01e00e94a40 not found: ID does not exist"
Sep 18 20:13:17.942678 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.942472    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"]
Sep 18 20:13:17.944506 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.944487    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg"]
Sep 18 20:13:17.952610 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:17.952585    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:18.021512 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.021475    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c03c40-f166-487e-9d3a-b5b351ada4d3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.021512 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.021510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.021769 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.021717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5824658-6c79-45da-a530-93bce89eb389-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.021769 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.021754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c03c40-f166-487e-9d3a-b5b351ada4d3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.021914 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.021803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4v6j\" (UniqueName: \"kubernetes.io/projected/e5824658-6c79-45da-a530-93bce89eb389-kube-api-access-g4v6j\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.022041 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.021953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.022211 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.022179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c03c40-f166-487e-9d3a-b5b351ada4d3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.022435 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.022386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.022556 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.022446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5824658-6c79-45da-a530-93bce89eb389-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.022556 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.022514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9kv\" (UniqueName: \"kubernetes.io/projected/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-api-access-8l9kv\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.022556 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.022550    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.022767 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.022745    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.023261 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.023239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.024819 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.024803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c03c40-f166-487e-9d3a-b5b351ada4d3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.029709 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.029689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9kv\" (UniqueName: \"kubernetes.io/projected/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-api-access-8l9kv\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-64b29\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.062588 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.062498    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:18.123778 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.123742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.123778 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.123787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5824658-6c79-45da-a530-93bce89eb389-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.124077 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.123809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.124077 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.123872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5824658-6c79-45da-a530-93bce89eb389-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.124077 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.123897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4v6j\" (UniqueName: \"kubernetes.io/projected/e5824658-6c79-45da-a530-93bce89eb389-kube-api-access-g4v6j\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.124282 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.124247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5824658-6c79-45da-a530-93bce89eb389-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.124532 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.124512    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.124532 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.124534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.126343 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.126320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5824658-6c79-45da-a530-93bce89eb389-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.131817 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.131799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4v6j\" (UniqueName: \"kubernetes.io/projected/e5824658-6c79-45da-a530-93bce89eb389-kube-api-access-g4v6j\") pod \"isvc-canary-stop-predictor-5d5b899946-9xkt2\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.145839 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.145814    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:18.184777 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.184738    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:18.417682 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.417653    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"]
Sep 18 20:13:18.419623 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:13:18.419592    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c03c40_f166_487e_9d3a_b5b351ada4d3.slice/crio-b3bc8df1359db370514586eca18ce190fcf7e1ea1380741269ee4bf86fe6d8f0 WatchSource:0}: Error finding container b3bc8df1359db370514586eca18ce190fcf7e1ea1380741269ee4bf86fe6d8f0: Status 404 returned error can't find the container with id b3bc8df1359db370514586eca18ce190fcf7e1ea1380741269ee4bf86fe6d8f0
Sep 18 20:13:18.472637 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.472603    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"]
Sep 18 20:13:18.479094 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:13:18.479046    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5824658_6c79_45da_a530_93bce89eb389.slice/crio-5840dbcb50dce0a9742ae015f86ed3f14d3543ab175cdde07044a65a019d16dc WatchSource:0}: Error finding container 5840dbcb50dce0a9742ae015f86ed3f14d3543ab175cdde07044a65a019d16dc: Status 404 returned error can't find the container with id 5840dbcb50dce0a9742ae015f86ed3f14d3543ab175cdde07044a65a019d16dc
Sep 18 20:13:18.928800 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.928762    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc70fe2f-f0e3-4bc8-b015-a01622a5592a" path="/var/lib/kubelet/pods/dc70fe2f-f0e3-4bc8-b015-a01622a5592a/volumes"
Sep 18 20:13:18.929240 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.929215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" event={"ID":"e5824658-6c79-45da-a530-93bce89eb389","Type":"ContainerStarted","Data":"3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09"}
Sep 18 20:13:18.929360 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.929282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" event={"ID":"e5824658-6c79-45da-a530-93bce89eb389","Type":"ContainerStarted","Data":"5840dbcb50dce0a9742ae015f86ed3f14d3543ab175cdde07044a65a019d16dc"}
Sep 18 20:13:18.929360 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.929320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" event={"ID":"89c03c40-f166-487e-9d3a-b5b351ada4d3","Type":"ContainerStarted","Data":"d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05"}
Sep 18 20:13:18.929360 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:18.929353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" event={"ID":"89c03c40-f166-487e-9d3a-b5b351ada4d3","Type":"ContainerStarted","Data":"b3bc8df1359db370514586eca18ce190fcf7e1ea1380741269ee4bf86fe6d8f0"}
Sep 18 20:13:22.947753 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:22.947719    2560 generic.go:356] "Generic (PLEG): container finished" podID="e5824658-6c79-45da-a530-93bce89eb389" containerID="3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09" exitCode=0
Sep 18 20:13:22.948322 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:22.947796    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" event={"ID":"e5824658-6c79-45da-a530-93bce89eb389","Type":"ContainerDied","Data":"3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09"}
Sep 18 20:13:22.949374 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:22.949346    2560 generic.go:356] "Generic (PLEG): container finished" podID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerID="d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05" exitCode=0
Sep 18 20:13:22.949480 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:22.949406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" event={"ID":"89c03c40-f166-487e-9d3a-b5b351ada4d3","Type":"ContainerDied","Data":"d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05"}
Sep 18 20:13:23.808060 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.808026    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:23.957218 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.957184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" event={"ID":"e5824658-6c79-45da-a530-93bce89eb389","Type":"ContainerStarted","Data":"893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed"}
Sep 18 20:13:23.957690 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.957256    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" event={"ID":"e5824658-6c79-45da-a530-93bce89eb389","Type":"ContainerStarted","Data":"9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e"}
Sep 18 20:13:23.957690 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.957570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:23.957812 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.957732    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:23.959105 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.959071    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:13:23.959512 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.959493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" event={"ID":"89c03c40-f166-487e-9d3a-b5b351ada4d3","Type":"ContainerStarted","Data":"604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e"}
Sep 18 20:13:23.959574 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.959540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" event={"ID":"89c03c40-f166-487e-9d3a-b5b351ada4d3","Type":"ContainerStarted","Data":"527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5"}
Sep 18 20:13:23.959862 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.959833    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:23.959929 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.959896    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:23.961047 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.961024    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:13:23.978309 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.978253    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podStartSLOduration=3.111068615 podStartE2EDuration="6.978238004s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867169389s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:23.977164665 +0000 UTC m=+2111.579527847" watchObservedRunningTime="2026-09-18 20:13:23.978238004 +0000 UTC m=+2111.580601186"
Sep 18 20:13:23.995956 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:23.995907    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podStartSLOduration=2.991864316 podStartE2EDuration="6.995890193s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.004025877s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:23.993689176 +0000 UTC m=+2111.596052382" watchObservedRunningTime="2026-09-18 20:13:23.995890193 +0000 UTC m=+2111.598253374"
Sep 18 20:13:24.922718 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:24.922685    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:13:24.922923 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:13:24.922902    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:13:24.970858 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:24.970811    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:13:24.971225 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:24.970833    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:13:29.972738 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:29.972708    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:13:29.973256 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:29.973233    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:13:29.973317 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:29.973290    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:13:29.973755 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:29.973734    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:13:36.305732 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:36.305692    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:36.364616 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:36.364582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:39.922365 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:39.922335    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:13:39.922729 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:13:39.922492    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:13:39.973368 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:39.973327    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:13:39.973753 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:39.973717    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:13:47.304932 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:47.304891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:49.973353 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:49.973314    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:13:49.973757 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:49.973594    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:13:54.922760 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:54.922723    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:13:54.923251 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:13:54.922959    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:13:59.973639 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:59.973590    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:13:59.974146 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:13:59.973600    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:14:09.922063 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:09.922026    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:14:09.922457 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:09.922221    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:14:09.974351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:09.974309    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:14:09.974542 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:09.974312    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:14:13.452866 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:13.452810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:18.598764 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:18.598729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:18.826071 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:18.826032    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:19.973518 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:19.973478    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:14:19.973974 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:19.973646    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:14:22.924525 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:22.924483    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:14:22.924950 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:22.924808    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:14:29.974251 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:29.974213    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:14:29.974707 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:29.974214    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:14:31.923012 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:31.922971    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:14:36.922387 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:36.922351    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:14:36.922777 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:36.922556    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:14:36.922777 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:36.922545    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:14:41.924064 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:41.924028    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:14:42.048092 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:42.048055    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"]
Sep 18 20:14:42.048367 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:42.048309    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container" containerID="cri-o://9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e" gracePeriod=30
Sep 18 20:14:42.048367 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:42.048354    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kube-rbac-proxy" containerID="cri-o://893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed" gracePeriod=30
Sep 18 20:14:42.049683 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:42.049663    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:14:42.323812 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:42.323724    2560 generic.go:356] "Generic (PLEG): container finished" podID="e5824658-6c79-45da-a530-93bce89eb389" containerID="893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed" exitCode=2
Sep 18 20:14:42.323812 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:42.323793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" event={"ID":"e5824658-6c79-45da-a530-93bce89eb389","Type":"ContainerDied","Data":"893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed"}
Sep 18 20:14:44.968959 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:44.968911    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 18 20:14:46.987740 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:46.987711    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:14:47.065776 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.065665    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5824658-6c79-45da-a530-93bce89eb389-kserve-provision-location\") pod \"e5824658-6c79-45da-a530-93bce89eb389\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") "
Sep 18 20:14:47.065776 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.065715    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5824658-6c79-45da-a530-93bce89eb389-proxy-tls\") pod \"e5824658-6c79-45da-a530-93bce89eb389\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") "
Sep 18 20:14:47.065776 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.065734    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-cabundle-cert\") pod \"e5824658-6c79-45da-a530-93bce89eb389\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") "
Sep 18 20:14:47.066125 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.065784    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4v6j\" (UniqueName: \"kubernetes.io/projected/e5824658-6c79-45da-a530-93bce89eb389-kube-api-access-g4v6j\") pod \"e5824658-6c79-45da-a530-93bce89eb389\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") "
Sep 18 20:14:47.066125 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.065839    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-kube-rbac-proxy-sar-config\") pod \"e5824658-6c79-45da-a530-93bce89eb389\" (UID: \"e5824658-6c79-45da-a530-93bce89eb389\") "
Sep 18 20:14:47.066125 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.066066    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5824658-6c79-45da-a530-93bce89eb389-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5824658-6c79-45da-a530-93bce89eb389" (UID: "e5824658-6c79-45da-a530-93bce89eb389"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:47.066247 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.066172    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5824658-6c79-45da-a530-93bce89eb389" (UID: "e5824658-6c79-45da-a530-93bce89eb389"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:47.066287 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.066250    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5824658-6c79-45da-a530-93bce89eb389" (UID: "e5824658-6c79-45da-a530-93bce89eb389"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:47.068003 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.067976    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5824658-6c79-45da-a530-93bce89eb389-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5824658-6c79-45da-a530-93bce89eb389" (UID: "e5824658-6c79-45da-a530-93bce89eb389"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:47.068101 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.068080    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5824658-6c79-45da-a530-93bce89eb389-kube-api-access-g4v6j" (OuterVolumeSpecName: "kube-api-access-g4v6j") pod "e5824658-6c79-45da-a530-93bce89eb389" (UID: "e5824658-6c79-45da-a530-93bce89eb389"). InnerVolumeSpecName "kube-api-access-g4v6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:47.166480 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.166445    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5824658-6c79-45da-a530-93bce89eb389-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:47.166480 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.166475    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5824658-6c79-45da-a530-93bce89eb389-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:47.166480 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.166486    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:47.166480 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.166494    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4v6j\" (UniqueName: \"kubernetes.io/projected/e5824658-6c79-45da-a530-93bce89eb389-kube-api-access-g4v6j\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:47.166749 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.166502    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5824658-6c79-45da-a530-93bce89eb389-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:47.347883 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.347770    2560 generic.go:356] "Generic (PLEG): container finished" podID="e5824658-6c79-45da-a530-93bce89eb389" containerID="9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e" exitCode=0
Sep 18 20:14:47.347883 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.347819    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"
Sep 18 20:14:47.347883 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.347871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" event={"ID":"e5824658-6c79-45da-a530-93bce89eb389","Type":"ContainerDied","Data":"9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e"}
Sep 18 20:14:47.348111 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.347934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2" event={"ID":"e5824658-6c79-45da-a530-93bce89eb389","Type":"ContainerDied","Data":"5840dbcb50dce0a9742ae015f86ed3f14d3543ab175cdde07044a65a019d16dc"}
Sep 18 20:14:47.348111 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.347962    2560 scope.go:117] "RemoveContainer" containerID="893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed"
Sep 18 20:14:47.357168 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.357150    2560 scope.go:117] "RemoveContainer" containerID="9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e"
Sep 18 20:14:47.364426 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.364408    2560 scope.go:117] "RemoveContainer" containerID="3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09"
Sep 18 20:14:47.368509 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.368486    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"]
Sep 18 20:14:47.375185 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.375098    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2"]
Sep 18 20:14:47.375254 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.375205    2560 scope.go:117] "RemoveContainer" containerID="893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed"
Sep 18 20:14:47.375534 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:47.375515    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed\": container with ID starting with 893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed not found: ID does not exist" containerID="893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed"
Sep 18 20:14:47.375603 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.375539    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed"} err="failed to get container status \"893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed\": rpc error: code = NotFound desc = could not find container \"893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed\": container with ID starting with 893f995c654ce6f023c5bfb501891d37dd7b8cb48804eb408522d369cf762fed not found: ID does not exist"
Sep 18 20:14:47.375603 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.375560    2560 scope.go:117] "RemoveContainer" containerID="9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e"
Sep 18 20:14:47.375824 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:47.375804    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e\": container with ID starting with 9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e not found: ID does not exist" containerID="9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e"
Sep 18 20:14:47.375888 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.375833    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e"} err="failed to get container status \"9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e\": rpc error: code = NotFound desc = could not find container \"9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e\": container with ID starting with 9499092ecbde33f5a047f2d9f57e10748004f4ac416708a9ab43c854e734866e not found: ID does not exist"
Sep 18 20:14:47.375888 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.375871    2560 scope.go:117] "RemoveContainer" containerID="3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09"
Sep 18 20:14:47.376151 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:47.376133    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09\": container with ID starting with 3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09 not found: ID does not exist" containerID="3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09"
Sep 18 20:14:47.376206 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.376156    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09"} err="failed to get container status \"3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09\": rpc error: code = NotFound desc = could not find container \"3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09\": container with ID starting with 3c7402a00d55f9bf6369b95beae5582cf41b06db5c44695665ee14bd449f1c09 not found: ID does not exist"
Sep 18 20:14:47.793744 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.793710    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"]
Sep 18 20:14:47.794057 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.794011    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" containerID="cri-o://527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5" gracePeriod=30
Sep 18 20:14:47.794171 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.794069    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kube-rbac-proxy" containerID="cri-o://604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e" gracePeriod=30
Sep 18 20:14:47.838635 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.838584    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:47.892385 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:47.892345    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:48.354283 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:48.354249    2560 generic.go:356] "Generic (PLEG): container finished" podID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerID="604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e" exitCode=2
Sep 18 20:14:48.354685 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:48.354319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" event={"ID":"89c03c40-f166-487e-9d3a-b5b351ada4d3","Type":"ContainerDied","Data":"604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e"}
Sep 18 20:14:48.922864 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:48.922827    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:14:48.923071 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:48.923035    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:14:48.928079 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:48.928047    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5824658-6c79-45da-a530-93bce89eb389" path="/var/lib/kubelet/pods/e5824658-6c79-45da-a530-93bce89eb389/volumes"
Sep 18 20:14:49.968377 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:49.968332    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 18 20:14:51.923641 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:51.923592    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:14:52.737750 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.737724    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:14:52.815680 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.815645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c03c40-f166-487e-9d3a-b5b351ada4d3-kserve-provision-location\") pod \"89c03c40-f166-487e-9d3a-b5b351ada4d3\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") "
Sep 18 20:14:52.815891 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.815714    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-cabundle-cert\") pod \"89c03c40-f166-487e-9d3a-b5b351ada4d3\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") "
Sep 18 20:14:52.815891 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.815734    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l9kv\" (UniqueName: \"kubernetes.io/projected/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-api-access-8l9kv\") pod \"89c03c40-f166-487e-9d3a-b5b351ada4d3\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") "
Sep 18 20:14:52.815891 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.815754    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c03c40-f166-487e-9d3a-b5b351ada4d3-proxy-tls\") pod \"89c03c40-f166-487e-9d3a-b5b351ada4d3\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") "
Sep 18 20:14:52.815891 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.815777    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-rbac-proxy-sar-config\") pod \"89c03c40-f166-487e-9d3a-b5b351ada4d3\" (UID: \"89c03c40-f166-487e-9d3a-b5b351ada4d3\") "
Sep 18 20:14:52.816122 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.816031    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c03c40-f166-487e-9d3a-b5b351ada4d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89c03c40-f166-487e-9d3a-b5b351ada4d3" (UID: "89c03c40-f166-487e-9d3a-b5b351ada4d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:52.816183 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.816144    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89c03c40-f166-487e-9d3a-b5b351ada4d3" (UID: "89c03c40-f166-487e-9d3a-b5b351ada4d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:52.816183 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.816166    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89c03c40-f166-487e-9d3a-b5b351ada4d3" (UID: "89c03c40-f166-487e-9d3a-b5b351ada4d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:52.818035 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.818013    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-api-access-8l9kv" (OuterVolumeSpecName: "kube-api-access-8l9kv") pod "89c03c40-f166-487e-9d3a-b5b351ada4d3" (UID: "89c03c40-f166-487e-9d3a-b5b351ada4d3"). InnerVolumeSpecName "kube-api-access-8l9kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:52.818112 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.818044    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c03c40-f166-487e-9d3a-b5b351ada4d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89c03c40-f166-487e-9d3a-b5b351ada4d3" (UID: "89c03c40-f166-487e-9d3a-b5b351ada4d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:52.916636 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.916599    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.916636 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.916633    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l9kv\" (UniqueName: \"kubernetes.io/projected/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-api-access-8l9kv\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.916636 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.916644    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c03c40-f166-487e-9d3a-b5b351ada4d3-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.916636 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.916653    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c03c40-f166-487e-9d3a-b5b351ada4d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.916964 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:52.916661    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c03c40-f166-487e-9d3a-b5b351ada4d3-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:14:53.378010 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.377909    2560 generic.go:356] "Generic (PLEG): container finished" podID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerID="527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5" exitCode=0
Sep 18 20:14:53.378010 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.377981    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" event={"ID":"89c03c40-f166-487e-9d3a-b5b351ada4d3","Type":"ContainerDied","Data":"527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5"}
Sep 18 20:14:53.378010 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.377985    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"
Sep 18 20:14:53.378534 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.378051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29" event={"ID":"89c03c40-f166-487e-9d3a-b5b351ada4d3","Type":"ContainerDied","Data":"b3bc8df1359db370514586eca18ce190fcf7e1ea1380741269ee4bf86fe6d8f0"}
Sep 18 20:14:53.378534 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.378078    2560 scope.go:117] "RemoveContainer" containerID="604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e"
Sep 18 20:14:53.386511 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.386494    2560 scope.go:117] "RemoveContainer" containerID="527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5"
Sep 18 20:14:53.394494 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.394471    2560 scope.go:117] "RemoveContainer" containerID="d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05"
Sep 18 20:14:53.394926 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.394906    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"]
Sep 18 20:14:53.398165 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.398141    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29"]
Sep 18 20:14:53.402976 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.402956    2560 scope.go:117] "RemoveContainer" containerID="604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e"
Sep 18 20:14:53.403288 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:53.403268    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e\": container with ID starting with 604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e not found: ID does not exist" containerID="604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e"
Sep 18 20:14:53.403333 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.403299    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e"} err="failed to get container status \"604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e\": rpc error: code = NotFound desc = could not find container \"604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e\": container with ID starting with 604f05b90ec19d5155af4a93f3c5bee9ff334b6dc54a34efa00e2e88773eca8e not found: ID does not exist"
Sep 18 20:14:53.403333 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.403319    2560 scope.go:117] "RemoveContainer" containerID="527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5"
Sep 18 20:14:53.403579 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:53.403563    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5\": container with ID starting with 527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5 not found: ID does not exist" containerID="527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5"
Sep 18 20:14:53.403617 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.403583    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5"} err="failed to get container status \"527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5\": rpc error: code = NotFound desc = could not find container \"527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5\": container with ID starting with 527421f6fbf40a582bc31f31ef656a5a9544b386ead610ec5802be0711b11ba5 not found: ID does not exist"
Sep 18 20:14:53.403617 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.403596    2560 scope.go:117] "RemoveContainer" containerID="d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05"
Sep 18 20:14:53.403789 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:14:53.403775    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05\": container with ID starting with d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05 not found: ID does not exist" containerID="d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05"
Sep 18 20:14:53.403827 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:53.403793    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05"} err="failed to get container status \"d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05\": rpc error: code = NotFound desc = could not find container \"d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05\": container with ID starting with d47df1819aa41ae98b1b337b3e3c9c0f84ac91058cf098099dabb522a5f3cf05 not found: ID does not exist"
Sep 18 20:14:54.927699 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:54.927666    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" path="/var/lib/kubelet/pods/89c03c40-f166-487e-9d3a-b5b351ada4d3/volumes"
Sep 18 20:14:58.027537 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.027490    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:58.049242 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049205    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"]
Sep 18 20:14:58.049534 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049522    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049536    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049545    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kube-rbac-proxy"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049551    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kube-rbac-proxy"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049559    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049566    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049577    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="storage-initializer"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049582    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="storage-initializer"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049589    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kube-rbac-proxy"
Sep 18 20:14:58.049590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049594    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kube-rbac-proxy"
Sep 18 20:14:58.049934 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049607    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="storage-initializer"
Sep 18 20:14:58.049934 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049612    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="storage-initializer"
Sep 18 20:14:58.049934 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049659    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kube-rbac-proxy"
Sep 18 20:14:58.049934 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049667    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kube-rbac-proxy"
Sep 18 20:14:58.049934 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049674    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5824658-6c79-45da-a530-93bce89eb389" containerName="kserve-container"
Sep 18 20:14:58.049934 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.049681    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c03c40-f166-487e-9d3a-b5b351ada4d3" containerName="kserve-container"
Sep 18 20:14:58.054414 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.054390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.056833 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.056813    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:14:58.056985 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.056960    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 18 20:14:58.057040 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.056991    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:14:58.057040 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.057027    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:14:58.057836 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.057823    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:14:58.057909 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.057870    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:14:58.057956 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.057935    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 18 20:14:58.062563 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.062539    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"]
Sep 18 20:14:58.159908 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.159844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrxk\" (UniqueName: \"kubernetes.io/projected/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-api-access-hdrxk\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.160109 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.159952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d47e6d9-5a57-4348-ab7e-e80d04129711-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.160109 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.159979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d47e6d9-5a57-4348-ab7e-e80d04129711-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.160109 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.160002    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.160109 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.160040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.176351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.176316    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:58.260714 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.260675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrxk\" (UniqueName: \"kubernetes.io/projected/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-api-access-hdrxk\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.260959 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.260754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d47e6d9-5a57-4348-ab7e-e80d04129711-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.260959 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.260792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d47e6d9-5a57-4348-ab7e-e80d04129711-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.260959 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.260816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.260959 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.260844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.261271 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.261246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d47e6d9-5a57-4348-ab7e-e80d04129711-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.261652 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.261629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.261732 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.261640    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.263330 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.263313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d47e6d9-5a57-4348-ab7e-e80d04129711-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.268835 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.268816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrxk\" (UniqueName: \"kubernetes.io/projected/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-api-access-hdrxk\") pod \"isvc-lightgbm-predictor-54f85ddf88-nzvr5\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.365755 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.365667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:14:58.636047 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:58.636012    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"]
Sep 18 20:14:58.638309 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:14:58.638279    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d47e6d9_5a57_4348_ab7e_e80d04129711.slice/crio-6355b08c0adb0ea83985141233843c7b44eaf17559bd04baf1156e441c2a8e72 WatchSource:0}: Error finding container 6355b08c0adb0ea83985141233843c7b44eaf17559bd04baf1156e441c2a8e72: Status 404 returned error can't find the container with id 6355b08c0adb0ea83985141233843c7b44eaf17559bd04baf1156e441c2a8e72
Sep 18 20:14:59.407864 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:59.407816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" event={"ID":"8d47e6d9-5a57-4348-ab7e-e80d04129711","Type":"ContainerStarted","Data":"d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7"}
Sep 18 20:14:59.408309 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:14:59.407898    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" event={"ID":"8d47e6d9-5a57-4348-ab7e-e80d04129711","Type":"ContainerStarted","Data":"6355b08c0adb0ea83985141233843c7b44eaf17559bd04baf1156e441c2a8e72"}
Sep 18 20:15:02.923911 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:02.923883    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:15:02.924314 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:15:02.924076    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:15:03.427143 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:03.427110    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerID="d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7" exitCode=0
Sep 18 20:15:03.427305 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:03.427184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" event={"ID":"8d47e6d9-5a57-4348-ab7e-e80d04129711","Type":"ContainerDied","Data":"d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7"}
Sep 18 20:15:03.433644 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:03.433501    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:15:08.689162 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:08.689120    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:10.464016 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:10.463976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" event={"ID":"8d47e6d9-5a57-4348-ab7e-e80d04129711","Type":"ContainerStarted","Data":"b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc"}
Sep 18 20:15:10.464422 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:10.464046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" event={"ID":"8d47e6d9-5a57-4348-ab7e-e80d04129711","Type":"ContainerStarted","Data":"cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4"}
Sep 18 20:15:10.464422 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:10.464243    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:15:10.483760 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:10.483705    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podStartSLOduration=2.194126823 podStartE2EDuration="12.483688643s" totalImagesPullingTime="6.434355729s" totalInitContainerRuntime="3.855206091s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:10.481723069 +0000 UTC m=+2218.084086251" watchObservedRunningTime="2026-09-18 20:15:10.483688643 +0000 UTC m=+2218.086051824"
Sep 18 20:15:11.468432 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:11.468398    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:15:11.469940 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:11.469910    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:15:12.473578 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:12.473537    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:15:13.817296 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:13.817261    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:13.970211 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:13.970175    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:17.478794 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:17.478764    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:15:17.479326 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:17.479299    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:15:17.922276 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:17.922247    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:15:17.922459 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:15:17.922436    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:15:27.479786 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:27.479738    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:15:31.922762 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:31.922720    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:15:31.923257 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:15:31.922983    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:15:37.479474 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:37.479388    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:15:44.922161 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:44.922120    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:15:44.922603 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:15:44.922345    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:15:47.480192 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:47.480148    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:15:57.479992 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:57.479941    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:15:57.922150 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:15:57.922123    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:15:57.922323 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:15:57.922307    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:16:03.974963 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:03.974931    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:07.479480 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:07.479440    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:16:08.927508 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:08.927479    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:16:08.927993 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:16:08.927676    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:16:09.090609 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:09.090572    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:09.298177 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:09.298085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:17.479916 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:17.479868    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:16:20.922135 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:20.922097    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:16:20.922531 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:16:20.922299    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:16:27.479781 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:27.479735    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:16:31.922514 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:31.922477    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:16:31.922951 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:16:31.922663    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:16:37.480067 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:37.480015    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:16:41.922785 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:41.922733    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:16:45.922483 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:45.922448    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:16:45.922997 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:16:45.922624    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:16:51.924606 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:51.924569    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:16:58.598782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:58.598743    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"]
Sep 18 20:16:58.599308 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:58.599019    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" containerID="cri-o://cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4" gracePeriod=30
Sep 18 20:16:58.599308 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:58.599113    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kube-rbac-proxy" containerID="cri-o://b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc" gracePeriod=30
Sep 18 20:16:58.620411 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:58.620383    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:58.943913 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:58.943882    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerID="b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc" exitCode=2
Sep 18 20:16:58.944085 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:16:58.943932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" event={"ID":"8d47e6d9-5a57-4348-ab7e-e80d04129711","Type":"ContainerDied","Data":"b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc"}
Sep 18 20:17:00.921977 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:00.921944    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:17:00.922379 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:17:00.922152    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:17:01.922682 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:01.922619    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:17:02.473937 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:02.473898    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 18 20:17:03.744452 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:03.744419    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:03.846612 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:03.846574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:04.541097 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.541069    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:17:04.624239 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.624203    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d47e6d9-5a57-4348-ab7e-e80d04129711-proxy-tls\") pod \"8d47e6d9-5a57-4348-ab7e-e80d04129711\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") "
Sep 18 20:17:04.624445 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.624251    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-cabundle-cert\") pod \"8d47e6d9-5a57-4348-ab7e-e80d04129711\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") "
Sep 18 20:17:04.624445 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.624275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdrxk\" (UniqueName: \"kubernetes.io/projected/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-api-access-hdrxk\") pod \"8d47e6d9-5a57-4348-ab7e-e80d04129711\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") "
Sep 18 20:17:04.624445 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.624298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-rbac-proxy-sar-config\") pod \"8d47e6d9-5a57-4348-ab7e-e80d04129711\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") "
Sep 18 20:17:04.624445 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.624364    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d47e6d9-5a57-4348-ab7e-e80d04129711-kserve-provision-location\") pod \"8d47e6d9-5a57-4348-ab7e-e80d04129711\" (UID: \"8d47e6d9-5a57-4348-ab7e-e80d04129711\") "
Sep 18 20:17:04.624720 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.624695    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d47e6d9-5a57-4348-ab7e-e80d04129711" (UID: "8d47e6d9-5a57-4348-ab7e-e80d04129711"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:04.624768 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.624712    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d47e6d9-5a57-4348-ab7e-e80d04129711-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d47e6d9-5a57-4348-ab7e-e80d04129711" (UID: "8d47e6d9-5a57-4348-ab7e-e80d04129711"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:04.624768 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.624706    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d47e6d9-5a57-4348-ab7e-e80d04129711" (UID: "8d47e6d9-5a57-4348-ab7e-e80d04129711"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:04.626462 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.626433    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-api-access-hdrxk" (OuterVolumeSpecName: "kube-api-access-hdrxk") pod "8d47e6d9-5a57-4348-ab7e-e80d04129711" (UID: "8d47e6d9-5a57-4348-ab7e-e80d04129711"). InnerVolumeSpecName "kube-api-access-hdrxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:04.626571 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.626545    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d47e6d9-5a57-4348-ab7e-e80d04129711-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d47e6d9-5a57-4348-ab7e-e80d04129711" (UID: "8d47e6d9-5a57-4348-ab7e-e80d04129711"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:04.725538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.725424    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d47e6d9-5a57-4348-ab7e-e80d04129711-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:17:04.725538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.725460    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d47e6d9-5a57-4348-ab7e-e80d04129711-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:17:04.725538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.725472    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:17:04.725538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.725480    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdrxk\" (UniqueName: \"kubernetes.io/projected/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-api-access-hdrxk\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:17:04.725538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.725488    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d47e6d9-5a57-4348-ab7e-e80d04129711-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:17:04.970827 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.970795    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerID="cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4" exitCode=0
Sep 18 20:17:04.971273 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.970864    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"
Sep 18 20:17:04.971273 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.970872    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" event={"ID":"8d47e6d9-5a57-4348-ab7e-e80d04129711","Type":"ContainerDied","Data":"cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4"}
Sep 18 20:17:04.971273 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.970938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5" event={"ID":"8d47e6d9-5a57-4348-ab7e-e80d04129711","Type":"ContainerDied","Data":"6355b08c0adb0ea83985141233843c7b44eaf17559bd04baf1156e441c2a8e72"}
Sep 18 20:17:04.971273 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.970966    2560 scope.go:117] "RemoveContainer" containerID="b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc"
Sep 18 20:17:04.979380 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.979315    2560 scope.go:117] "RemoveContainer" containerID="cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4"
Sep 18 20:17:04.987337 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.987317    2560 scope.go:117] "RemoveContainer" containerID="d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7"
Sep 18 20:17:04.988666 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.988643    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"]
Sep 18 20:17:04.993962 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.993931    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5"]
Sep 18 20:17:04.995549 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.995535    2560 scope.go:117] "RemoveContainer" containerID="b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc"
Sep 18 20:17:04.995964 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:17:04.995844    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc\": container with ID starting with b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc not found: ID does not exist" containerID="b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc"
Sep 18 20:17:04.996187 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.995970    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc"} err="failed to get container status \"b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc\": rpc error: code = NotFound desc = could not find container \"b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc\": container with ID starting with b4e7c3e115418c1afadb02bab8161a01b85ea2e90b86326e47b6137c243017bc not found: ID does not exist"
Sep 18 20:17:04.996187 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.995991    2560 scope.go:117] "RemoveContainer" containerID="cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4"
Sep 18 20:17:04.996313 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:17:04.996269    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4\": container with ID starting with cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4 not found: ID does not exist" containerID="cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4"
Sep 18 20:17:04.996313 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.996289    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4"} err="failed to get container status \"cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4\": rpc error: code = NotFound desc = could not find container \"cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4\": container with ID starting with cb06155e5d0a9ea327a32b5facdae6c3f7f7f18229699425c22bb33716c893c4 not found: ID does not exist"
Sep 18 20:17:04.996313 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.996303    2560 scope.go:117] "RemoveContainer" containerID="d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7"
Sep 18 20:17:04.996594 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:17:04.996571    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7\": container with ID starting with d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7 not found: ID does not exist" containerID="d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7"
Sep 18 20:17:04.996656 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:04.996602    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7"} err="failed to get container status \"d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7\": rpc error: code = NotFound desc = could not find container \"d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7\": container with ID starting with d6a69f463870cc4a2dab449f090fb4c57989bce44a9fbd09e6c8fa1cbfaccdd7 not found: ID does not exist"
Sep 18 20:17:06.928448 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:06.928413    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" path="/var/lib/kubelet/pods/8d47e6d9-5a57-4348-ab7e-e80d04129711/volumes"
Sep 18 20:17:14.922943 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:14.922903    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:17:16.021642 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:16.021596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"}
Sep 18 20:17:16.022049 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:16.021816    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:17:39.861777 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:39.861743    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:44.996631 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:44.996583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:45.544366 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:45.544330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:47.028909 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:17:47.028874    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:18:45.126016 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:45.125939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:46.495286 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:46.495247    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:50.232188 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.232155    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:50.265923 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.265888    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"]
Sep 18 20:18:50.266209 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.266197    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="storage-initializer"
Sep 18 20:18:50.266276 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.266212    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="storage-initializer"
Sep 18 20:18:50.266276 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.266228    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kube-rbac-proxy"
Sep 18 20:18:50.266276 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.266238    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kube-rbac-proxy"
Sep 18 20:18:50.266276 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.266255    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container"
Sep 18 20:18:50.266276 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.266262    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container"
Sep 18 20:18:50.266452 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.266351    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kserve-container"
Sep 18 20:18:50.266452 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.266365    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d47e6d9-5a57-4348-ab7e-e80d04129711" containerName="kube-rbac-proxy"
Sep 18 20:18:50.269760 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.269733    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.272132 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.272103    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:18:50.272294 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.272213    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 18 20:18:50.272294 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.272271    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:18:50.272294 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.272280    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:18:50.273002 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.272986    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 18 20:18:50.273122 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.273005    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:18:50.273122 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.273084    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:18:50.278464 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.278439    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"]
Sep 18 20:18:50.343114 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.343073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:50.370481 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.370436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/431c11f8-1e8a-49d1-8f8e-1ddaec094253-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.370904 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.370519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.370904 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.370566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.370904 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.370587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.370904 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.370640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsxl\" (UniqueName: \"kubernetes.io/projected/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-api-access-cbsxl\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.471177 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.471131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsxl\" (UniqueName: \"kubernetes.io/projected/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-api-access-cbsxl\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.471371 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.471236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/431c11f8-1e8a-49d1-8f8e-1ddaec094253-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.471371 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.471261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.471371 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.471280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.471371 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.471304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.471784 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.471762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.471983 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.471963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.472088 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.472070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.473794 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.473774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/431c11f8-1e8a-49d1-8f8e-1ddaec094253-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.478929 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.478905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsxl\" (UniqueName: \"kubernetes.io/projected/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-api-access-cbsxl\") pod \"isvc-xgboost-v2-predictor-fcc668bc-hm8pd\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.581888 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.581784    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:18:50.855901 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:50.855873    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"]
Sep 18 20:18:50.857461 ip-10-0-142-19 kubenswrapper[2560]: W0918 20:18:50.857427    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431c11f8_1e8a_49d1_8f8e_1ddaec094253.slice/crio-393abc0e20c4f787d8a7d002e7ab1aa12b1aa8d6155b1f5d790338f9cbed40e7 WatchSource:0}: Error finding container 393abc0e20c4f787d8a7d002e7ab1aa12b1aa8d6155b1f5d790338f9cbed40e7: Status 404 returned error can't find the container with id 393abc0e20c4f787d8a7d002e7ab1aa12b1aa8d6155b1f5d790338f9cbed40e7
Sep 18 20:18:51.440459 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:51.440418    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" event={"ID":"431c11f8-1e8a-49d1-8f8e-1ddaec094253","Type":"ContainerStarted","Data":"50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395"}
Sep 18 20:18:51.440989 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:51.440488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" event={"ID":"431c11f8-1e8a-49d1-8f8e-1ddaec094253","Type":"ContainerStarted","Data":"393abc0e20c4f787d8a7d002e7ab1aa12b1aa8d6155b1f5d790338f9cbed40e7"}
Sep 18 20:18:55.143473 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:55.143361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:55.458451 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:55.458416    2560 generic.go:356] "Generic (PLEG): container finished" podID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerID="50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395" exitCode=0
Sep 18 20:18:55.458631 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:18:55.458492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" event={"ID":"431c11f8-1e8a-49d1-8f8e-1ddaec094253","Type":"ContainerDied","Data":"50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395"}
Sep 18 20:19:00.254091 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:00.254053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:00.524762 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:00.524665    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:13.643435 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:13.643316    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:14.559987 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:14.559951    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" event={"ID":"431c11f8-1e8a-49d1-8f8e-1ddaec094253","Type":"ContainerStarted","Data":"95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99"}
Sep 18 20:19:14.560186 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:14.560014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" event={"ID":"431c11f8-1e8a-49d1-8f8e-1ddaec094253","Type":"ContainerStarted","Data":"ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e"}
Sep 18 20:19:14.560245 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:14.560202    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:19:14.579396 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:14.579339    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podStartSLOduration=2.333406102 podStartE2EDuration="24.579318929s" totalImagesPullingTime="18.380168382s" totalInitContainerRuntime="3.865744445s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:14.577439371 +0000 UTC m=+2462.179802556" watchObservedRunningTime="2026-09-18 20:19:14.579318929 +0000 UTC m=+2462.181682110"
Sep 18 20:19:15.564954 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:15.564869    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:19:15.566227 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:15.566198    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:19:16.569669 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:16.569630    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:19:17.783406 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:17.783368    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:21.574794 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:21.574757    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:19:21.575325 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:21.575296    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:19:31.575650 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:31.575606    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:19:33.644274 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:33.644239    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925" exitCode=1
Sep 18 20:19:33.644707 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:33.644293    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"}
Sep 18 20:19:33.644707 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:33.644348    2560 scope.go:117] "RemoveContainer" containerID="5b551ff1f9f52261bacb9a6808dcce86644fdd5d384667d8683ac3abf70c6517"
Sep 18 20:19:33.644707 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:33.644668    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:19:33.644931 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:19:33.644900    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:19:37.027493 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:37.027455    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:19:37.027910 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:37.027836    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:19:37.028070 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:19:37.028048    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:19:41.575280 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:41.575237    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:19:42.047009 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:42.046971    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:19:42.047338 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:42.047324    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:19:42.047519 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:19:42.047500    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:19:50.365424 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:50.365382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:51.575345 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:51.575297    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:19:52.924823 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:52.924791    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:19:52.925217 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:19:52.925013    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:19:55.495181 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:55.495138    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:55.740103 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:55.740067    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:59.170980 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:19:59.170944    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:01.576206 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:01.576163    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:20:06.922958 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:06.922874    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:20:06.923332 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:20:06.923069    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:20:11.575261 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:11.575225    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:20:19.922312 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:19.922276    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:20:19.922703 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:20:19.922468    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:20:21.575909 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:21.575842    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:20:25.638988 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:25.638944    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:30.785698 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:30.785664    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:30.952481 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:30.952443    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:31.576481 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:31.576443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:20:32.923896 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:32.923866    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:20:32.924288 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:20:32.924057    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:20:40.981324 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:40.981285    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"]
Sep 18 20:20:40.981720 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:40.981559    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" containerID="cri-o://ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e" gracePeriod=30
Sep 18 20:20:40.981720 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:40.981608    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kube-rbac-proxy" containerID="cri-o://95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99" gracePeriod=30
Sep 18 20:20:41.003803 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:41.003771    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:41.570370 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:41.570325    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 18 20:20:41.575497 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:41.575452    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:20:41.959864 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:41.959816    2560 generic.go:356] "Generic (PLEG): container finished" podID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerID="95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99" exitCode=2
Sep 18 20:20:41.960053 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:41.959896    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" event={"ID":"431c11f8-1e8a-49d1-8f8e-1ddaec094253","Type":"ContainerDied","Data":"95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99"}
Sep 18 20:20:44.923106 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:44.923062    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:20:44.923513 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:20:44.923301    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:20:45.337887 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.337840    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:20:45.354087 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.354046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/431c11f8-1e8a-49d1-8f8e-1ddaec094253-proxy-tls\") pod \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") "
Sep 18 20:20:45.354263 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.354105    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kserve-provision-location\") pod \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") "
Sep 18 20:20:45.354263 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.354142    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-rbac-proxy-sar-config\") pod \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") "
Sep 18 20:20:45.354263 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.354220    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-cabundle-cert\") pod \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") "
Sep 18 20:20:45.354263 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.354244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbsxl\" (UniqueName: \"kubernetes.io/projected/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-api-access-cbsxl\") pod \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\" (UID: \"431c11f8-1e8a-49d1-8f8e-1ddaec094253\") "
Sep 18 20:20:45.354558 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.354467    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "431c11f8-1e8a-49d1-8f8e-1ddaec094253" (UID: "431c11f8-1e8a-49d1-8f8e-1ddaec094253"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:45.354625 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.354585    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "431c11f8-1e8a-49d1-8f8e-1ddaec094253" (UID: "431c11f8-1e8a-49d1-8f8e-1ddaec094253"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:20:45.354625 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.354605    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "431c11f8-1e8a-49d1-8f8e-1ddaec094253" (UID: "431c11f8-1e8a-49d1-8f8e-1ddaec094253"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:20:45.356746 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.356693    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-api-access-cbsxl" (OuterVolumeSpecName: "kube-api-access-cbsxl") pod "431c11f8-1e8a-49d1-8f8e-1ddaec094253" (UID: "431c11f8-1e8a-49d1-8f8e-1ddaec094253"). InnerVolumeSpecName "kube-api-access-cbsxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:20:45.356991 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.356872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431c11f8-1e8a-49d1-8f8e-1ddaec094253-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "431c11f8-1e8a-49d1-8f8e-1ddaec094253" (UID: "431c11f8-1e8a-49d1-8f8e-1ddaec094253"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:20:45.454825 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.454729    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-cabundle-cert\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:20:45.454825 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.454764    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsxl\" (UniqueName: \"kubernetes.io/projected/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-api-access-cbsxl\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:20:45.454825 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.454775    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/431c11f8-1e8a-49d1-8f8e-1ddaec094253-proxy-tls\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:20:45.454825 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.454786    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kserve-provision-location\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:20:45.454825 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.454794    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/431c11f8-1e8a-49d1-8f8e-1ddaec094253-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-19.ec2.internal\" DevicePath \"\""
Sep 18 20:20:45.984664 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.984622    2560 generic.go:356] "Generic (PLEG): container finished" podID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerID="ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e" exitCode=0
Sep 18 20:20:45.985088 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.984688    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"
Sep 18 20:20:45.985088 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.984713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" event={"ID":"431c11f8-1e8a-49d1-8f8e-1ddaec094253","Type":"ContainerDied","Data":"ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e"}
Sep 18 20:20:45.985088 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.984801    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd" event={"ID":"431c11f8-1e8a-49d1-8f8e-1ddaec094253","Type":"ContainerDied","Data":"393abc0e20c4f787d8a7d002e7ab1aa12b1aa8d6155b1f5d790338f9cbed40e7"}
Sep 18 20:20:45.985088 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.984831    2560 scope.go:117] "RemoveContainer" containerID="95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99"
Sep 18 20:20:45.993860 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:45.993830    2560 scope.go:117] "RemoveContainer" containerID="ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e"
Sep 18 20:20:46.001889 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.001869    2560 scope.go:117] "RemoveContainer" containerID="50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395"
Sep 18 20:20:46.005170 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.005143    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"]
Sep 18 20:20:46.008700 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.008674    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd"]
Sep 18 20:20:46.010386 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.010365    2560 scope.go:117] "RemoveContainer" containerID="95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99"
Sep 18 20:20:46.010688 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:20:46.010669    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99\": container with ID starting with 95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99 not found: ID does not exist" containerID="95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99"
Sep 18 20:20:46.010737 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.010698    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99"} err="failed to get container status \"95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99\": rpc error: code = NotFound desc = could not find container \"95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99\": container with ID starting with 95072c2399c7b5fd3a08d8248c73f99984a6ed70872215f80aa48a2995431f99 not found: ID does not exist"
Sep 18 20:20:46.010737 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.010716    2560 scope.go:117] "RemoveContainer" containerID="ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e"
Sep 18 20:20:46.010988 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:20:46.010969    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e\": container with ID starting with ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e not found: ID does not exist" containerID="ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e"
Sep 18 20:20:46.011033 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.010995    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e"} err="failed to get container status \"ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e\": rpc error: code = NotFound desc = could not find container \"ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e\": container with ID starting with ea5ebaabd81c410e9bbd8bc9fde42cc087e33963226c95c11ebc39186a02712e not found: ID does not exist"
Sep 18 20:20:46.011033 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.011011    2560 scope.go:117] "RemoveContainer" containerID="50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395"
Sep 18 20:20:46.011240 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:20:46.011223    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395\": container with ID starting with 50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395 not found: ID does not exist" containerID="50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395"
Sep 18 20:20:46.011284 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.011244    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395"} err="failed to get container status \"50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395\": rpc error: code = NotFound desc = could not find container \"50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395\": container with ID starting with 50a5706e1105615cd30a4442e83ecd8cbd034901e6289ab66e586f093dbab395 not found: ID does not exist"
Sep 18 20:20:46.234603 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.234569    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:46.308590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.308508    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:46.927331 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:46.927302    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" path="/var/lib/kubelet/pods/431c11f8-1e8a-49d1-8f8e-1ddaec094253/volumes"
Sep 18 20:20:55.922364 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:20:55.922327    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:20:55.923060 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:20:55.922530    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:21:09.922032 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:21:09.921998    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:21:09.922426 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:21:09.922208    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:21:20.885535 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:21:20.885499    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:22.924309 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:21:22.924273    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:21:22.924713 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:21:22.924529    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:21:26.020864 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:21:26.020816    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:26.416283 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:21:26.416245    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:35.922106 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:21:35.922020    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:21:35.922563 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:21:35.922222    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:21:48.922905 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:21:48.922831    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:21:48.923400 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:21:48.923103    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:22:03.922138 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:03.922101    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:22:03.922545 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:22:03.922269    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:22:14.922749 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:14.922709    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:22:14.923272 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:22:14.922937    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:22:16.351840 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:16.351803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:16.489975 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:16.489942    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:16.667467 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:16.667432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:25.922352 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:25.922320    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:22:25.922760 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:22:25.922512    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:22:37.922522 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:37.922480    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:22:37.923043 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:22:37.922699    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:22:46.100525 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:46.100493    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:49.922474 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:49.922432    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:22:49.922959 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:22:49.922644    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:22:51.229271 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:51.229239    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:51.373466 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:22:51.373404    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:00.922071 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:00.922033    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:23:00.922565 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:23:00.922235    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:23:14.922474 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:14.922392    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:23:14.922900 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:23:14.922560    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:23:29.922320 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:29.922286    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:23:29.922799 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:23:29.922467    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:23:36.523015 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:36.522978    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:37.579561 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:37.579527    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:38.334151 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:38.334117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:40.923028 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:40.922994    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:23:40.923414 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:23:40.923189    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:23:47.596265 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:47.596233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:48.744976 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:48.744925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:48.893653 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:48.893617    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:51.922894 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:23:51.922835    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:23:51.923278 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:23:51.923054    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:24:02.685162 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:02.685129    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:03.922090 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:03.922051    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:24:03.922500 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:24:03.922222    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:24:07.791748 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:07.791715    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:07.993687 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:07.993654    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:16.922867 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:16.922830    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:24:16.925362 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:24:16.923072    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:24:31.922187 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:31.922152    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:24:31.922599 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:24:31.922327    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:24:46.922623 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:46.922533    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:24:46.923106 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:46.922953    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:24:48.045036 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:48.045002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"}
Sep 18 20:24:48.045425 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:24:48.045243    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:25:09.259168 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:09.259125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:14.393817 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:14.393778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:15.061155 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:15.061119    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:18.766541 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:18.766504    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:19.052290 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:19.052212    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:25:19.833294 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:19.833262    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:19.937831 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:19.937796    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:29.842590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:29.842556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:30.920006 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:30.919973    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:31.104385 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:31.104353    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:39.350807 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:25:39.350776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:16.066681 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:26:16.066601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:21.191842 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:26:21.191796    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:21.290986 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:26:21.290949    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:50.965735 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:26:50.965705    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:52.147472 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:26:52.147434    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:52.247609 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:26:52.247507    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:02.037614 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:02.037581    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:03.157717 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:03.157679    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:03.614462 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:03.614385    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:04.050561 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:04.050518    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": dial tcp 10.133.0.26:8081: connect: connection refused"
Sep 18 20:27:04.648804 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:04.648773    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcbf5392-7320-4796-b1df-099b7b688384" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b" exitCode=1
Sep 18 20:27:04.649278 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:04.648823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerDied","Data":"418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"}
Sep 18 20:27:04.649278 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:04.648907    2560 scope.go:117] "RemoveContainer" containerID="2e243192ab71ef8eefb0342ffcd0d2bc8387d0767d351c4aa73b48020d481925"
Sep 18 20:27:04.649278 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:04.649235    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:27:04.649464 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:27:04.649442    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:27:09.049546 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:09.049506    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:27:09.049961 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:09.049897    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:27:09.050088 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:27:09.050066    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:27:12.047129 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:12.047087    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:27:12.047537 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:12.047486    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:27:12.047663 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:27:12.047643    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:27:26.923070 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:26.923032    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:27:26.923570 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:27:26.923239    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:27:39.922838 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:39.922757    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:27:39.923228 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:27:39.922973    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:27:54.923047 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:27:54.923008    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:27:54.923470 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:27:54.923174    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:28:02.786118 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:02.786081    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:06.922530 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:06.922493    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:28:06.922958 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:28:06.922671    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:28:07.894876 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:07.894830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:08.074288 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:08.074254    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:20.922778 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:20.922736    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:28:20.923196 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:28:20.922957    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:28:34.922862 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:34.922822    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:28:34.923384 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:28:34.923056    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:28:45.668866 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:45.668820    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:47.922273 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:47.922243    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:28:47.922667 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:28:47.922406    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:28:55.245137 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:55.245103    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:59.922008 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:28:59.921974    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:28:59.922411 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:28:59.922171    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:29:11.922971 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:29:11.922932    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:29:11.923457 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:29:11.923107    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:29:24.922388 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:29:24.922346    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:29:24.924740 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:29:24.922522    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:29:25.064839 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:29:25.064803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:38.922384 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:29:38.922350    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:29:38.922784 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:29:38.922552    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:29:49.922464 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:29:49.922430    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:29:49.923066 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:29:49.922624    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:29:59.673207 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:29:59.673169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:00.922681 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:30:00.922650    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:30:00.923113 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:30:00.922833    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:30:04.791863 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:30:04.791821    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:05.218282 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:30:05.218241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:13.922784 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:30:13.922753    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:30:13.923188 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:30:13.922985    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:30:24.922118 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:30:24.922074    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:30:24.922610 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:30:24.922358    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:30:39.922254 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:30:39.922173    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:30:39.922631 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:30:39.922359    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:30:54.922351 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:30:54.922317    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:30:54.922755 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:30:54.922493    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:31:06.922223 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:31:06.922184    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:31:06.922817 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:31:06.922364    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:31:21.922370 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:31:21.922338    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:31:21.922757 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:31:21.922500    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:31:36.922580 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:31:36.922541    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:31:36.923127 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:31:36.922715    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:31:48.922033 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:31:48.921994    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:31:48.922640 ip-10-0-142-19 kubenswrapper[2560]: E0918 20:31:48.922183    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-2kd9d_kserve(dcbf5392-7320-4796-b1df-099b7b688384)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" podUID="dcbf5392-7320-4796-b1df-099b7b688384"
Sep 18 20:31:50.020763 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:31:50.020730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:50.160645 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:31:50.160610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:03.922126 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:03.922093    2560 scope.go:117] "RemoveContainer" containerID="418a6578858cfa113ed333eb28d3cc8c73d2bcf4caa88df874036729dc0dfd2b"
Sep 18 20:32:03.922490 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:03.922390    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:32:04.071279 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:04.071248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:04.962356 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:04.962322    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d" event={"ID":"dcbf5392-7320-4796-b1df-099b7b688384","Type":"ContainerStarted","Data":"7010bb31d63056043173d9c6107879d7477c2e4c69c87ea92552c7f4e39eeb10"}
Sep 18 20:32:04.962733 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:04.962526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:32:05.319264 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:05.319144    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:11.250170 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:11.250132    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:11.299353 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:11.299317    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:16.255353 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:16.255311    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:35.968737 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:35.968706    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-2kd9d"
Sep 18 20:32:44.742540 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:44.742499    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:44.875649 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:44.875615    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:44.972131 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:44.972094    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:47.782316 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:47.782267    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d579p_d6972273-7829-45b0-9158-bfada0b133e1/global-pull-secret-syncer/0.log"
Sep 18 20:32:47.859940 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:47.859908    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:47.931291 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:47.931260    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log"
Sep 18 20:32:47.958971 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:47.958925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:48.061826 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:48.061738    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-19.ec2.internal_580e69a6077123655fe2b32c49b4aa69/haproxy/0.log"
Sep 18 20:32:50.385902 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:50.385832    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:50.446350 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:50.446314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2crvg_2bee0e71-2a3f-4d61-9c19-d02dc7e9cca1/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:52.358622 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:52.358541    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86559f9ff-qw6qq_dce2c667-177e-4a95-b724-a53790a29278/metrics-server/0.log"
Sep 18 20:32:52.398646 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:52.398604    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-gjd5j_3e8c7eeb-ce95-4728-a639-81882f2b95b9/monitoring-plugin/0.log"
Sep 18 20:32:52.448539 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:52.448506    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdnj8_efeb9794-9860-4b1b-8877-eef83329d23b/node-exporter/0.log"
Sep 18 20:32:52.487442 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:52.487400    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdnj8_efeb9794-9860-4b1b-8877-eef83329d23b/kube-rbac-proxy/0.log"
Sep 18 20:32:52.524186 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:52.524157    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdnj8_efeb9794-9860-4b1b-8877-eef83329d23b/init-textfile/0.log"
Sep 18 20:32:52.771320 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:52.771287    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kct9k_9e06902f-e997-4b56-987a-7fe7a5eb96cf/kube-rbac-proxy-main/0.log"
Sep 18 20:32:52.805445 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:52.805410    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kct9k_9e06902f-e997-4b56-987a-7fe7a5eb96cf/kube-rbac-proxy-self/0.log"
Sep 18 20:32:52.842664 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:52.842632    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kct9k_9e06902f-e997-4b56-987a-7fe7a5eb96cf/openshift-state-metrics/0.log"
Sep 18 20:32:53.046281 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:53.046191    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-bfjxg_f5275c10-10f9-457d-82bb-ee79691f6a5f/prometheus-operator/0.log"
Sep 18 20:32:53.085384 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:53.085354    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-bfjxg_f5275c10-10f9-457d-82bb-ee79691f6a5f/kube-rbac-proxy/0.log"
Sep 18 20:32:53.127313 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:53.127286    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-2tgcf_472ca6b6-4435-4e36-b0c7-19e1069d8d6d/prometheus-operator-admission-webhook/0.log"
Sep 18 20:32:53.167430 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:53.167403    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d996c5bc8-czwdk_1639cf01-6883-409f-876e-d7510722ee5d/telemeter-client/0.log"
Sep 18 20:32:53.261195 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:53.261163    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d996c5bc8-czwdk_1639cf01-6883-409f-876e-d7510722ee5d/reload/0.log"
Sep 18 20:32:53.292256 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:53.292226    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d996c5bc8-czwdk_1639cf01-6883-409f-876e-d7510722ee5d/kube-rbac-proxy/0.log"
Sep 18 20:32:55.786126 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:55.786062    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74969fdc99-5w84g_db887214-22a8-4b8a-b04f-c80a2e02d3aa/console/0.log"
Sep 18 20:32:55.830068 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:55.830034    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-btdg5_84ffd9ed-0328-4d69-bcdd-1e858ca1bcee/download-server/0.log"
Sep 18 20:32:56.248398 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248368    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"]
Sep 18 20:32:56.248731 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248719    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="storage-initializer"
Sep 18 20:32:56.248777 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248734    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="storage-initializer"
Sep 18 20:32:56.248777 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248756    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kube-rbac-proxy"
Sep 18 20:32:56.248777 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248761    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kube-rbac-proxy"
Sep 18 20:32:56.248777 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248771    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container"
Sep 18 20:32:56.248777 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248778    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container"
Sep 18 20:32:56.248980 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248823    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kserve-container"
Sep 18 20:32:56.248980 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.248832    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="431c11f8-1e8a-49d1-8f8e-1ddaec094253" containerName="kube-rbac-proxy"
Sep 18 20:32:56.252060 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.252038    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.254388 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.254370    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cdvqj\"/\"default-dockercfg-8c4rd\""
Sep 18 20:32:56.255300 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.255281    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cdvqj\"/\"kube-root-ca.crt\""
Sep 18 20:32:56.255370 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.255321    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cdvqj\"/\"openshift-service-ca.crt\""
Sep 18 20:32:56.262028 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.262003    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"]
Sep 18 20:32:56.331901 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.331834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-proc\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.331901 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.331903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-sys\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.332142 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.331935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-podres\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.332142 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.331977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-lib-modules\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.332142 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.332021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8htl\" (UniqueName: \"kubernetes.io/projected/752b21e4-fcc7-4d53-9900-0f38234e6416-kube-api-access-m8htl\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432537 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-proc\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432537 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-sys\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-podres\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-lib-modules\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8htl\" (UniqueName: \"kubernetes.io/projected/752b21e4-fcc7-4d53-9900-0f38234e6416-kube-api-access-m8htl\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-proc\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-sys\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-podres\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.432782 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.432714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752b21e4-fcc7-4d53-9900-0f38234e6416-lib-modules\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.440499 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.440468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8htl\" (UniqueName: \"kubernetes.io/projected/752b21e4-fcc7-4d53-9900-0f38234e6416-kube-api-access-m8htl\") pod \"perf-node-gather-daemonset-4w8rk\" (UID: \"752b21e4-fcc7-4d53-9900-0f38234e6416\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.562059 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.561966    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:56.834901 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:56.834874    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"]
Sep 18 20:32:57.195309 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:57.195273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk" event={"ID":"752b21e4-fcc7-4d53-9900-0f38234e6416","Type":"ContainerStarted","Data":"f2e3d4f9aa37d17d409e7f0c5c04bc4de441e4a65128fe4ec4361cfb88a14751"}
Sep 18 20:32:57.195492 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:57.195327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk" event={"ID":"752b21e4-fcc7-4d53-9900-0f38234e6416","Type":"ContainerStarted","Data":"b2d1518aa3dbc54ad4035ad14c205ec2aa65522adedc4d7cd1918ab73818599e"}
Sep 18 20:32:57.195492 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:57.195366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:32:57.210387 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:57.210332    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk" podStartSLOduration=1.210312287 podStartE2EDuration="1.210312287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:57.209630387 +0000 UTC m=+3284.811993570" watchObservedRunningTime="2026-09-18 20:32:57.210312287 +0000 UTC m=+3284.812675469"
Sep 18 20:32:57.287779 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:57.287743    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bttxn_77157d0e-1282-4af5-95c3-8a63614e1dbc/dns/0.log"
Sep 18 20:32:57.313281 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:57.313249    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bttxn_77157d0e-1282-4af5-95c3-8a63614e1dbc/kube-rbac-proxy/0.log"
Sep 18 20:32:57.475197 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:57.475120    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6gg2_d8cbf7a2-e0da-417e-8353-9397e53a62d0/dns-node-resolver/0.log"
Sep 18 20:32:57.986133 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:57.986097    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7qjmw_e4555a60-1138-42ec-8931-6b4f9091edc3/node-ca/0.log"
Sep 18 20:32:59.067538 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:59.067505    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-42fvk_95b30c4f-7580-44ba-a3f9-229180da41d5/serve-healthcheck-canary/0.log"
Sep 18 20:32:59.431384 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:59.431349    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48hgl_d37679c8-1c35-4052-942d-624cb5ebe193/kube-rbac-proxy/0.log"
Sep 18 20:32:59.450349 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:59.450322    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48hgl_d37679c8-1c35-4052-942d-624cb5ebe193/exporter/0.log"
Sep 18 20:32:59.469729 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:32:59.469703    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48hgl_d37679c8-1c35-4052-942d-624cb5ebe193/extractor/0.log"
Sep 18 20:33:01.990705 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:01.990669    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-b9b5d54f8-cgnvb_2020bd6f-aa3d-4420-903c-0171ef1672cf/manager/0.log"
Sep 18 20:33:02.015098 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:02.015042    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-2kd9d_dcbf5392-7320-4796-b1df-099b7b688384/manager/10.log"
Sep 18 20:33:02.016395 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:02.016366    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-2kd9d_dcbf5392-7320-4796-b1df-099b7b688384/manager/9.log"
Sep 18 20:33:02.558150 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:02.558111    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-w9lpg_0a4ca5c3-a9cf-4957-90b6-e29e91f9c80e/seaweedfs-tls-custom/0.log"
Sep 18 20:33:02.581973 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:02.581944    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-rnb82_de7e8ed5-55e5-4a2b-9104-46dfdb71530f/seaweedfs-tls-serving/0.log"
Sep 18 20:33:03.209663 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:03.209637    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-4w8rk"
Sep 18 20:33:08.171334 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.171298    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjk6b_5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6/kube-multus-additional-cni-plugins/0.log"
Sep 18 20:33:08.191734 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.191698    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjk6b_5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6/egress-router-binary-copy/0.log"
Sep 18 20:33:08.210531 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.210500    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjk6b_5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6/cni-plugins/0.log"
Sep 18 20:33:08.229016 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.228980    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjk6b_5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6/bond-cni-plugin/0.log"
Sep 18 20:33:08.248279 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.248249    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjk6b_5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6/routeoverride-cni/0.log"
Sep 18 20:33:08.267176 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.267150    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjk6b_5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6/whereabouts-cni-bincopy/0.log"
Sep 18 20:33:08.286590 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.286561    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjk6b_5f0e4816-d28a-4f5e-b22e-c0fe86bd51d6/whereabouts-cni/0.log"
Sep 18 20:33:08.329497 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.329465    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gsd9d_79834c8f-5e1f-4ddf-a2fc-0b393da1343b/kube-multus/0.log"
Sep 18 20:33:08.387368 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.387340    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6v4ch_d583d577-4430-4736-9cff-606fe6ee92e9/network-metrics-daemon/0.log"
Sep 18 20:33:08.404237 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:08.404205    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6v4ch_d583d577-4430-4736-9cff-606fe6ee92e9/kube-rbac-proxy/0.log"
Sep 18 20:33:09.797843 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:09.797807    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m748s_05f4058e-a32c-473e-b179-158080388120/ovn-controller/0.log"
Sep 18 20:33:09.816460 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:09.816434    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m748s_05f4058e-a32c-473e-b179-158080388120/ovn-acl-logging/0.log"
Sep 18 20:33:09.839309 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:09.839279    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m748s_05f4058e-a32c-473e-b179-158080388120/kube-rbac-proxy-node/0.log"
Sep 18 20:33:09.860477 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:09.860441    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m748s_05f4058e-a32c-473e-b179-158080388120/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 20:33:09.882162 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:09.882137    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m748s_05f4058e-a32c-473e-b179-158080388120/northd/0.log"
Sep 18 20:33:09.902508 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:09.902478    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m748s_05f4058e-a32c-473e-b179-158080388120/nbdb/0.log"
Sep 18 20:33:09.921392 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:09.921363    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m748s_05f4058e-a32c-473e-b179-158080388120/sbdb/0.log"
Sep 18 20:33:09.988833 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:09.988794    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m748s_05f4058e-a32c-473e-b179-158080388120/ovnkube-controller/0.log"
Sep 18 20:33:10.974224 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:10.974189    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qvw9h_e6c68d39-1394-4fea-969f-422a6817eb6b/network-check-target-container/0.log"
Sep 18 20:33:11.821949 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:11.821921    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2klft_4f33dd29-44ca-43b1-bfb0-0a38b154ee58/iptables-alerter/0.log"
Sep 18 20:33:12.514110 ip-10-0-142-19 kubenswrapper[2560]: I0918 20:33:12.514083    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-ddpkm_7bce6468-88f5-4663-988a-a81cfbb2a9aa/tuned/0.log"