Sep 16 18:40:46.767307 ip-10-0-143-155 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:47.109690 ip-10-0-143-155 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:47.109690 ip-10-0-143-155 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:47.109690 ip-10-0-143-155 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:47.109690 ip-10-0-143-155 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:47.109690 ip-10-0-143-155 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:47.110616 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.110527    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:47.116782 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116759    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:47.116782 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116777    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:47.116782 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116783    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:47.116782 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116787    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:47.116782 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116791    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116796    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116800    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116804    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116809    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116814    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116818    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116822    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116825    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116829    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116833    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116837    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116842    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116846    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116850    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116853    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116857    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116861    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116865    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116869    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:47.117179 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116873    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116877    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116881    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116885    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116889    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116893    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116897    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116900    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116905    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116909    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116913    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116917    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116921    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116925    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116929    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116934    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116939    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116945    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116949    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:47.118038 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116954    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116958    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116963    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116967    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116972    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116976    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116981    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116985    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116989    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116994    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.116999    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117022    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117027    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117031    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117036    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117044    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117050    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117056    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117060    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:47.118857 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117069    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117075    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117080    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117085    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117089    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117094    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117098    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117102    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117106    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117112    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117117    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117122    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117126    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117130    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117134    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117140    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117146    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117151    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117156    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117161    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:47.119669 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117167    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117171    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117175    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117180    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117186    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117190    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117197    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117202    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117206    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117210    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117215    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117219    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117223    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117228    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117233    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117237    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.117244    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118423    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118434    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118440    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:47.120560 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118445    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118450    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118456    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118461    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118466    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118471    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118475    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118479    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118484    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118489    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118493    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118498    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118502    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118507    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118511    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118515    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118519    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118524    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118528    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118533    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:47.121448 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118538    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118542    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118547    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118551    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118555    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118559    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118564    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118568    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118573    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118577    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118581    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118586    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118590    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118594    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118599    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118604    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118609    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118614    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118618    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:47.122116 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118623    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118627    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118631    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118636    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118640    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118644    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118648    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118653    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118657    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118661    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118665    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118669    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118674    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118678    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118684    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118692    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118698    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118703    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118707    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118713    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:47.122693 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118718    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118723    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118727    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118731    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118735    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118739    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118744    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118749    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118753    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118761    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118768    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118773    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118777    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118782    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118787    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118791    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118796    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118801    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118806    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118810    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:47.123287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118817    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118821    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118827    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118832    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118836    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118840    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118845    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118850    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118856    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118861    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118866    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118871    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118875    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118879    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118883    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118887    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.118892    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119516    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119526    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119536    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:47.123879 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119543    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119550    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119556    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119561    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119566    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119571    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119576    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119580    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119586    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119592    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119597    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119602    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119607    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119612    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119617    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119622    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119627    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119631    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119636    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:47.124710 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119640    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119645    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119649    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119654    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119658    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119663    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119667    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119671    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119675    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119679    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119683    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119688    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119695    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119699    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119704    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119710    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119714    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119719    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119723    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119727    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:47.125267 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119732    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119736    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119740    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119745    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119749    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119754    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119758    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119762    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119766    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119770    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119775    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119779    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119783    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119787    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119792    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119796    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119801    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119805    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119810    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:47.125889 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119814    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119818    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119822    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119826    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119831    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119837    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119841    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119845    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119850    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119855    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119859    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119863    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119869    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119874    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119879    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119883    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119887    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119892    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119896    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119901    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:47.126384 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119905    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119909    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119913    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119917    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119922    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119926    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119930    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119935    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119939    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119944    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119948    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119952    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119956    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119961    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119965    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119969    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119974    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.119980    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120111    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120168    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120181    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:47.126917 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120190    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120198    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120204    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120211    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120217    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120223    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120228    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120233    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120239    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120244    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120249    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120254    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120259    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120271    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120277    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120284    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120289    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120295    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120304    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120310    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120317    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120322    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120327    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:47.127458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120333    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120338    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120343    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120347    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120353    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120361    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120369    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120374    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120379    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120384    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120389    2578 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120395    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120404    2578 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120410    2578 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120415    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120420    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120425    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120431    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120436    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120441    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120446    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120451    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120457    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120462    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120467    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:47.128033 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120471    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120476    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120483    2578 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120490    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120495    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120500    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120505    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120510    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120515    2578 flags.go:64] FLAG: --help="false"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120520    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120525    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120530    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120537    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120543    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120549    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120554    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120558    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120563    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120568    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120573    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120579    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120584    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120589    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120594    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:47.128719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120599    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120604    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120609    2578 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120615    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120620    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120625    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120634    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120639    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120644    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120648    2578 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120654    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120659    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120664    2578 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120669    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120677    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120682    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120689    2578 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120694    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120699    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120704    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120709    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120714    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120718    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120723    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120736    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:47.129348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120741    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120746    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120751    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120761    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120771    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120776    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120782    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120787    2578 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120792    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120796    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c56bf98bdce7b738"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120801    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120807    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120812    2578 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120817    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120823    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120828    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120833    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120839    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120846    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120851    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120857    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120862    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120867    2578 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120872    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120878    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:47.129963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120882    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120888    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120893    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120899    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120904    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120909    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120915    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120919    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120924    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120929    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120934    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120940    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120946    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120955    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120960    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120964    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120971    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120976    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120981    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120986    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120991    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.120997    2578 flags.go:64] FLAG: --v="2"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.121023    2578 flags.go:64] FLAG: --version="false"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.121030    2578 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.121043    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:47.130648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.121049    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121247    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121254    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121259    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121265    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121269    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121274    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121278    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121282    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121287    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121291    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121296    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121300    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121305    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121309    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121314    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121318    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121322    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121328    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:47.131293 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121335    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121341    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121346    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121350    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121354    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121359    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121363    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121368    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121373    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121377    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121382    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121386    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121390    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121395    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121399    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121403    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121408    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121414    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121422    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:47.131773 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121428    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121433    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121437    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121442    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121447    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121451    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121456    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121460    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121465    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121469    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121473    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121478    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121484    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121488    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121494    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121498    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121503    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121508    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121512    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:47.132287 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121517    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121524    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121530    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121535    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121539    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121544    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121548    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121552    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121556    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121560    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121565    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121570    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121574    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121579    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121583    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121587    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121592    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121596    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121600    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121604    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:47.132747 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121609    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121613    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121617    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121622    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121626    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121630    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121634    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121638    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121644    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121649    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121653    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121658    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121662    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121668    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121675    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121679    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121683    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121687    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121692    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121696    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:47.133268 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121701    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121705    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.121709    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.122307    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.130456    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.130476    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.131210    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.131218    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131294    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131299    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131303    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131306    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131309    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131312    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131315    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:47.133797 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131318    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131321    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131324    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131326    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131329    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131331    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131334    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131337    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131339    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131342    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131345    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131349    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131353    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131355    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131359    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131362    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131365    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131367    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131370    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:47.134254 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131372    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131375    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131378    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131381    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131384    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131388    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131390    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131393    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131396    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131399    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131401    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131403    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131406    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131409    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131411    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131414    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131416    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131419    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131421    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131424    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131426    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:47.134753 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131429    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131432    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131434    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131437    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131439    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131442    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131445    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131447    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131450    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131453    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131456    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131458    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131461    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131463    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131466    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131470    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131473    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131475    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131478    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131480    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:47.135318 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131483    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131485    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131488    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131490    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131493    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131496    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131498    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131501    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131503    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131506    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131508    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131511    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131513    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131516    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131518    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131520    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131523    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131525    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131528    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131531    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:47.135807 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131533    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131536    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131538    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131541    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131545    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131549    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131553    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131557    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131561    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131565    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131567    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131570    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.131576    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131717    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:47.136337 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131723    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131727    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131730    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131733    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131736    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131739    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131741    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131744    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131746    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131749    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131751    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131754    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131757    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131760    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131762    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131765    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131767    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131770    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131773    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131775    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:47.136699 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131778    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131781    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131783    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131786    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131788    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131791    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131794    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131797    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131801    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131805    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131807    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131811    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131814    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131817    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131820    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131822    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131825    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131828    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131830    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131833    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:47.137215 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131836    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131838    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131841    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131844    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131846    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131849    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131851    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131854    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131856    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131859    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131861    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131864    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131866    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131868    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131871    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131873    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131876    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131878    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131881    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:47.137740 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131883    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131887    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131889    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131891    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131894    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131896    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131899    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131902    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131904    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131907    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131909    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131912    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131914    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131917    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131919    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131922    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131924    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131928    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131931    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131934    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:47.138223 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131936    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131939    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131942    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131944    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131946    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131949    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131951    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131954    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131956    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131959    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131961    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131964    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131967    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131969    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131973    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131977    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131980    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131985    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:47.138712 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:47.131988    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:47.139441 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.131993    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:47.139441 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.132153    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:47.139441 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.134186    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:47.139441 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.135001    2578 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:47.139441 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.135118    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:47.139441 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.135678    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:47.139441 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.137567    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:47.141460 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.141435    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:47.150530 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.150512    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:47.154945 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.154931    2578 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:47.155878 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.155863    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:47.160826 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.160800    2578 fs.go:135] Filesystem UUIDs: map[07098d84-33cb-4d50-89fe-05ebe6b7f7b7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b8d4fa9c-f866-4e1a-a556-e2d41b8b8b93:/dev/nvme0n1p4]
Sep 16 18:40:47.160925 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.160825    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:47.165792 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.165667    2578 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:47.164600679 +0000 UTC m=+0.295147387 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099794 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec260e0cd80243dc6d2b29657d99c224 SystemUUID:ec260e0c-d802-43dc-6d2b-29657d99c224 BootID:bc5f02e6-0b24-4d9c-81c0-198cb385d43e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:b3:c9:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:b3:c9:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:f7:5a:38:d2:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:47.165792 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.165783    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:47.165967 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.165897    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:47.168572 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.168545    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:47.168716 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.168574    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:47.169204 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.169194    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:47.169238 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.169208    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:47.169238 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.169216    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:47.169416 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.169397    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:47.169879 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.169867    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:47.170900 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.170889    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:47.174124 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174111    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:47.174254 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174244    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:47.174285 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174281    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:47.174319 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174298    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:47.174319 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174314    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:47.174373 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174323    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:47.174373 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174333    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:47.174739 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174728    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:47.174785 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.174747    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:47.177223 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.177208    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:47.179662 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.179645    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:47.179751 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.179684    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:47.181407 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181389    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:47.181482 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181416    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:47.181482 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181426    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:47.181482 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181435    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:47.181482 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181448    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:47.181482 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181457    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:47.181482 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181478    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:47.181649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181489    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:47.181649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181499    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:47.181649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181508    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:47.181649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181531    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:47.181649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.181540    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:47.182683 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.182604    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:47.182771 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.182745    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:47.185932 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.185911    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:47.186036 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.185943    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:47.186036 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.185984    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:47.187701 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.187687    2578 server.go:1268] "Started kubelet"
Sep 16 18:40:47.187763 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.187737    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:47.187866 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.187826    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:47.187914 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.187893    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:47.188768 ip-10-0-143-155 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:47.189439 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.188877    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:47.190284 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.190268    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:47.193363 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.192658    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-155.ec2.internal.18d5e135945303ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-155.ec2.internal,UID:ip-10-0-143-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-155.ec2.internal,},FirstTimestamp:2026-09-16 18:40:47.18762491 +0000 UTC m=+0.318171611,LastTimestamp:2026-09-16 18:40:47.18762491 +0000 UTC m=+0.318171611,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-155.ec2.internal,}"
Sep 16 18:40:47.194104 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.194074    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:47.196210 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.196192    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:47.196627 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.196614    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:47.196739 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.196716    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.197292 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197273    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:47.197369 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197274    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:47.197369 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197306    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:47.197652 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197370    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:47.197652 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197384    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:47.197752 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197676    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:47.197752 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197686    2578 factory.go:55] Registering systemd factory
Sep 16 18:40:47.197752 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197692    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:47.197936 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197922    2578 factory.go:153] Registering CRI-O factory
Sep 16 18:40:47.197936 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197937    2578 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:47.198052 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197955    2578 factory.go:103] Registering Raw factory
Sep 16 18:40:47.198052 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.197988    2578 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:47.198462 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.198449    2578 manager.go:349] Starting recovery of all containers
Sep 16 18:40:47.201457 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.201435    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:47.201457 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.201446    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:47.213543 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.213380    2578 manager.go:354] Recovery completed
Sep 16 18:40:47.217975 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.217960    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:47.220129 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.220103    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:47.220210 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.220137    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:47.220210 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.220149    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:47.220741 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.220725    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:47.220741 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.220734    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:47.220741 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.220738    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:47.220933 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.220758    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:47.221750 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.221736    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:47.221808 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.221766    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:47.221808 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.221779    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:47.222035 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.222021    2578 policy_none.go:49] "None policy: Start"
Sep 16 18:40:47.222105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.222042    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:47.222105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.222056    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:47.222777 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.222707    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-155.ec2.internal.18d5e1359642e766  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-155.ec2.internal,UID:ip-10-0-143-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-155.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-155.ec2.internal,},FirstTimestamp:2026-09-16 18:40:47.220123494 +0000 UTC m=+0.350670196,LastTimestamp:2026-09-16 18:40:47.220123494 +0000 UTC m=+0.350670196,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-155.ec2.internal,}"
Sep 16 18:40:47.223124 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.223111    2578 policy_none.go:47] "Start"
Sep 16 18:40:47.231898 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.231822    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-155.ec2.internal.18d5e1359643319a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-155.ec2.internal,UID:ip-10-0-143-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-155.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-155.ec2.internal,},FirstTimestamp:2026-09-16 18:40:47.22014249 +0000 UTC m=+0.350689192,LastTimestamp:2026-09-16 18:40:47.22014249 +0000 UTC m=+0.350689192,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-155.ec2.internal,}"
Sep 16 18:40:47.239837 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.239753    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-155.ec2.internal.18d5e13596435966  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-155.ec2.internal,UID:ip-10-0-143-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-155.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-155.ec2.internal,},FirstTimestamp:2026-09-16 18:40:47.220152678 +0000 UTC m=+0.350699381,LastTimestamp:2026-09-16 18:40:47.220152678 +0000 UTC m=+0.350699381,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-155.ec2.internal,}"
Sep 16 18:40:47.245583 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.245564    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nd6tn"
Sep 16 18:40:47.247282 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.247197    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-155.ec2.internal.18d5e1359642e766\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-155.ec2.internal.18d5e1359642e766  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-155.ec2.internal,UID:ip-10-0-143-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-155.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-155.ec2.internal,},FirstTimestamp:2026-09-16 18:40:47.220123494 +0000 UTC m=+0.350670196,LastTimestamp:2026-09-16 18:40:47.221749587 +0000 UTC m=+0.352296288,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-155.ec2.internal,}"
Sep 16 18:40:47.252645 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.252580    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-155.ec2.internal.18d5e1359643319a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-155.ec2.internal.18d5e1359643319a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-155.ec2.internal,UID:ip-10-0-143-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-155.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-155.ec2.internal,},FirstTimestamp:2026-09-16 18:40:47.22014249 +0000 UTC m=+0.350689192,LastTimestamp:2026-09-16 18:40:47.221771658 +0000 UTC m=+0.352318360,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-155.ec2.internal,}"
Sep 16 18:40:47.252742 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.252628    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nd6tn"
Sep 16 18:40:47.257856 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.257775    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-155.ec2.internal.18d5e13596435966\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-155.ec2.internal.18d5e13596435966  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-155.ec2.internal,UID:ip-10-0-143-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-155.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-155.ec2.internal,},FirstTimestamp:2026-09-16 18:40:47.220152678 +0000 UTC m=+0.350699381,LastTimestamp:2026-09-16 18:40:47.22178263 +0000 UTC m=+0.352329332,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-155.ec2.internal,}"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.266192    2578 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.266226    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.266236    2578 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.266576    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.266589    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.266690    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.266769    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.266779    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.267831    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:47.276491 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.267872    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.324398 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.324367    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:47.325898 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.325876    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:47.325989 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.325932    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:47.325989 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.325959    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:47.326087 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.326023    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:47.328740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.328720    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:47.366922 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.366825    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:47.367768 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.367746    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:47.367865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.367785    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:47.367865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.367800    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:47.367865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.367828    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.377269 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.377251    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.377322 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.377274    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-155.ec2.internal\": node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.393616 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.393584    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.427077 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.427052    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"]
Sep 16 18:40:47.427155 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.427120    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:47.428531 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.428516    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:47.428590 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.428549    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:47.428590 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.428559    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:47.430857 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.430844    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:47.431034 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.431020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.431084 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.431065    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:47.432801 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.432784    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:47.432889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.432813    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:47.432889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.432822    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:47.432889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.432786    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:47.432889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.432855    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:47.432889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.432867    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:47.435116 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.435101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.435199 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.435126    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:47.437023 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.436988    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:47.437100 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.437031    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:47.437100 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.437042    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:47.459786 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.459767    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-155.ec2.internal\" not found" node="ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.464231 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.464215    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-155.ec2.internal\" not found" node="ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.494355 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.494334    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.498736 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.498717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b899a3c315c0814e264cff323a4907d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-155.ec2.internal\" (UID: \"7b899a3c315c0814e264cff323a4907d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.498825 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.498742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee3faaf407844b52f4750d947306742-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"3ee3faaf407844b52f4750d947306742\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.498825 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.498760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee3faaf407844b52f4750d947306742-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"3ee3faaf407844b52f4750d947306742\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.594711 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.594682    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.598952 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.598932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b899a3c315c0814e264cff323a4907d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-155.ec2.internal\" (UID: \"7b899a3c315c0814e264cff323a4907d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.599023 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.598902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b899a3c315c0814e264cff323a4907d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-155.ec2.internal\" (UID: \"7b899a3c315c0814e264cff323a4907d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.599023 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.599002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee3faaf407844b52f4750d947306742-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"3ee3faaf407844b52f4750d947306742\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.599105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.599034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee3faaf407844b52f4750d947306742-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"3ee3faaf407844b52f4750d947306742\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.599105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.599059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee3faaf407844b52f4750d947306742-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"3ee3faaf407844b52f4750d947306742\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.599105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.599088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee3faaf407844b52f4750d947306742-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"3ee3faaf407844b52f4750d947306742\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.695462 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.695381    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.762898 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.762863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.768345 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:47.768079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:47.796184 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.796150    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.896710 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.896672    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:47.997107 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:47.997079    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:48.095076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.095043    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:48.097454 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:48.097432    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:48.134756 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.134725    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:48.135263 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.134914    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:48.135263 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.134923    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:48.198161 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:48.198135    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:48.258530 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.258438    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:47 +0000 UTC" deadline="2028-04-04 02:29:13.756564588 +0000 UTC"
Sep 16 18:40:48.258530 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.258480    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13567h48m25.498088121s"
Sep 16 18:40:48.298866 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:48.298836    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Sep 16 18:40:48.317457 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.317434    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:48.352900 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:48.352865    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b899a3c315c0814e264cff323a4907d.slice/crio-41bf8e8fab17175310cc8bf35da25c7fc61631fa828213560c02b32632bcb90f WatchSource:0}: Error finding container 41bf8e8fab17175310cc8bf35da25c7fc61631fa828213560c02b32632bcb90f: Status 404 returned error can't find the container with id 41bf8e8fab17175310cc8bf35da25c7fc61631fa828213560c02b32632bcb90f
Sep 16 18:40:48.353544 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:48.353523    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee3faaf407844b52f4750d947306742.slice/crio-1d31ab2d77dea5056d0b21c87c69007ce60a389c2f340715677d06dae2cc1c45 WatchSource:0}: Error finding container 1d31ab2d77dea5056d0b21c87c69007ce60a389c2f340715677d06dae2cc1c45: Status 404 returned error can't find the container with id 1d31ab2d77dea5056d0b21c87c69007ce60a389c2f340715677d06dae2cc1c45
Sep 16 18:40:48.357709 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.357680    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:48.397345 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.397320    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:48.407889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.407870    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:48.408768 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.408755    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Sep 16 18:40:48.419506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.419489    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:48.440568 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:48.440546    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:49.176212 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.175816    2578 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:49.185375 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.185343    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:49.185838 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.185811    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-pgrc2","openshift-network-operator/iptables-alerter-w88db","openshift-ovn-kubernetes/ovnkube-node-6jg58","kube-system/global-pull-secret-syncer-s25jm","kube-system/konnectivity-agent-8tlk8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w","openshift-cluster-node-tuning-operator/tuned-9csgf","openshift-dns/node-resolver-drqxf","openshift-image-registry/node-ca-q2cpl","openshift-multus/multus-additional-cni-plugins-xx92x","openshift-multus/network-metrics-daemon-wdr9l","kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal","openshift-multus/multus-t8sp4"]
Sep 16 18:40:49.188618 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.188593    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.191573 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.190951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:49.191573 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.191061    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:40:49.191573 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.191403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:49.191573 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.191416    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pw4tg\""
Sep 16 18:40:49.191880 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.191645    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:49.194049 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.193560    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.196290 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.196037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.196290 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.196135    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:40:49.196290 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.196242    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:49.196551 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.196323    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:49.196551 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.196415    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vg8fr\""
Sep 16 18:40:49.197288 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.197259    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:49.198230 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.198210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:49.198488 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.198426    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:49.198954 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.198753    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:49.199415 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.199396    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:49.201520 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.201233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:40:49.204111 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.202315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.204111 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.203641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.204282 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.204154    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tzrkn\""
Sep 16 18:40:49.204594 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.204575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:49.205722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.204906    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:49.205722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.204942    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:49.205722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.205276    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:49.205722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.205315    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:49.205722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.205493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbvkb\""
Sep 16 18:40:49.206248 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.206225    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysconfig\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-tmp\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-slash\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4n7j\" (UniqueName: \"kubernetes.io/projected/87a8beac-18a0-4743-9368-e0eea3507ad8-kube-api-access-s4n7j\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/310f9f3d-4268-4c5b-b046-afa2cebb3685-kubelet-config\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-sys-fs\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-systemd-units\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-systemd\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-var-lib-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87a8beac-18a0-4743-9368-e0eea3507ad8-ovn-node-metrics-cert\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/310f9f3d-4268-4c5b-b046-afa2cebb3685-dbus\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhhr\" (UniqueName: \"kubernetes.io/projected/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-kube-api-access-rwhhr\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68880f21-301d-4f7f-bb64-1f210f499111-serviceca\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.210043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-sys\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-tuned\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-cni-bin\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rbwd6\""
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-cni-netd\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-env-overrides\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-ovnkube-script-lib\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af08d1a0-b107-4438-b7ed-8061c969ba8d-konnectivity-ca\") pod \"konnectivity-agent-8tlk8\" (UID: \"af08d1a0-b107-4438-b7ed-8061c969ba8d\") " pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-socket-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-registration-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7tvv\" (UniqueName: \"kubernetes.io/projected/b19e5fc5-c807-4afb-a848-7cc107eb74a6-kube-api-access-r7tvv\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207905    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-kubernetes\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysctl-conf\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-kubelet\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.210848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.207997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-ovn\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-run\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-host\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-etc-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-log-socket\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-device-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68880f21-301d-4f7f-bb64-1f210f499111-host\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q84bp\" (UniqueName: \"kubernetes.io/projected/68880f21-301d-4f7f-bb64-1f210f499111-kube-api-access-q84bp\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-modprobe-d\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-systemd\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-lib-modules\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-var-lib-kubelet\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208414    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d82j\" (UniqueName: \"kubernetes.io/projected/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-kube-api-access-4d82j\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-run-netns\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysctl-d\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.211708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-node-log\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-ovnkube-config\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.208850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af08d1a0-b107-4438-b7ed-8061c969ba8d-agent-certs\") pod \"konnectivity-agent-8tlk8\" (UID: \"af08d1a0-b107-4438-b7ed-8061c969ba8d\") " pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.211192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.211239    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.211288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-777c7\""
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.211345    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:49.212534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.211532    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:49.213496 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.213475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.215783 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.215763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.216454 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.216432    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:49.216551 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.216453    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:49.216551 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.216495    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g89zc\""
Sep 16 18:40:49.216742 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.216723    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:49.218361 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.218341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:49.218468 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.218419    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:40:49.218579 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.218560    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:49.218740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.218703    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wmxd8\""
Sep 16 18:40:49.219818 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.219797    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:49.220198 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.220177    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:49.221214 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.221194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.223885 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.223749    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:49.224257 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.224226    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:49.224453 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.224438    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-42vhc\""
Sep 16 18:40:49.298914 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.298880    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:49.309797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.309768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q84bp\" (UniqueName: \"kubernetes.io/projected/68880f21-301d-4f7f-bb64-1f210f499111-kube-api-access-q84bp\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.309797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.309801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-systemd\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.310019 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.309829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-var-lib-kubelet\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.310019 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.309854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d82j\" (UniqueName: \"kubernetes.io/projected/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-kube-api-access-4d82j\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.310613 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.310588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-ovnkube-config\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.310673 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.310641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af08d1a0-b107-4438-b7ed-8061c969ba8d-agent-certs\") pod \"konnectivity-agent-8tlk8\" (UID: \"af08d1a0-b107-4438-b7ed-8061c969ba8d\") " pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:40:49.311218 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.311171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-systemd\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.311339 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.311274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-var-lib-kubelet\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.312076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.311486    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:49.312076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.311400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-cni-multus\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.312076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.311935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-conf-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.312076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.311975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-node-log\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.312076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-system-cni-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.312076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-tmp\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4n7j\" (UniqueName: \"kubernetes.io/projected/87a8beac-18a0-4743-9368-e0eea3507ad8-kube-api-access-s4n7j\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/310f9f3d-4268-4c5b-b046-afa2cebb3685-kubelet-config\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-sys-fs\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b708bf47-0a07-4e28-b464-487766a7377c-tmp-dir\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-cnibin\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-var-lib-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87a8beac-18a0-4743-9368-e0eea3507ad8-ovn-node-metrics-cert\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhhr\" (UniqueName: \"kubernetes.io/projected/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-kube-api-access-rwhhr\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.312452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-kubelet\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.313014 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-daemon-config\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.313014 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-ovnkube-config\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.313014 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-iptables-alerter-script\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.313014 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68880f21-301d-4f7f-bb64-1f210f499111-serviceca\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.313014 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-var-lib-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.313014 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-sys-fs\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.313014 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-cni-bin\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.313014 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.312714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-node-log\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.313457 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.313134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/310f9f3d-4268-4c5b-b046-afa2cebb3685-kubelet-config\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.313457 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.313254    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:49.313457 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.313321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.313457 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.313418    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret podName:310f9f3d-4268-4c5b-b046-afa2cebb3685 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:49.813393655 +0000 UTC m=+2.943940360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret") pod "global-pull-secret-syncer-s25jm" (UID: "310f9f3d-4268-4c5b-b046-afa2cebb3685") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:49.313706 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.313666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-cni-netd\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.313787 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.313732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-env-overrides\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.313787 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.313771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-ovnkube-script-lib\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.314484 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.314230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-env-overrides\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.314484 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.314277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.314484 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.314335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-cni-netd\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.314484 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.314449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68880f21-301d-4f7f-bb64-1f210f499111-serviceca\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.314744 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.314570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.314744 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.314698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-socket-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.314907 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.314884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-cni-bin\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.315062 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-socket-dir-parent\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.315146 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af08d1a0-b107-4438-b7ed-8061c969ba8d-konnectivity-ca\") pod \"konnectivity-agent-8tlk8\" (UID: \"af08d1a0-b107-4438-b7ed-8061c969ba8d\") " pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:40:49.315216 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-registration-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.315216 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87a8beac-18a0-4743-9368-e0eea3507ad8-ovnkube-script-lib\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.315373 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-registration-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.315373 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.315487 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gn9v\" (UniqueName: \"kubernetes.io/projected/b708bf47-0a07-4e28-b464-487766a7377c-kube-api-access-5gn9v\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.315487 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-cni-bin\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.315487 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b2x8\" (UniqueName: \"kubernetes.io/projected/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-kube-api-access-8b2x8\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.315645 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-host-slash\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.315645 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-kubelet\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.315796 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.315796 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.315796 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-ovn\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.315796 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b708bf47-0a07-4e28-b464-487766a7377c-hosts-file\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.315796 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-kubelet\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.315796 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-ovn\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.316139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-run\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.316139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af08d1a0-b107-4438-b7ed-8061c969ba8d-konnectivity-ca\") pod \"konnectivity-agent-8tlk8\" (UID: \"af08d1a0-b107-4438-b7ed-8061c969ba8d\") " pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:40:49.316139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-etc-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.316139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-run\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.316139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.315969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-log-socket\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.316139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-system-cni-dir\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.316139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-log-socket\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.316139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-etc-openvswitch\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68880f21-301d-4f7f-bb64-1f210f499111-host\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-modprobe-d\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68880f21-301d-4f7f-bb64-1f210f499111-host\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-lib-modules\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-run-netns\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-modprobe-d\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2m4j\" (UniqueName: \"kubernetes.io/projected/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-kube-api-access-b2m4j\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-lib-modules\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysctl-d\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-run-netns\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysctl-d\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.316537 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.317182 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrls\" (UniqueName: \"kubernetes.io/projected/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-kube-api-access-8vrls\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.317182 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.316995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.317182 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-hostroot\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.317182 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-multus-certs\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.317182 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysconfig\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.317475 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysconfig\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.317475 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-slash\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.317475 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cnibin\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.317475 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:49.317475 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-cni-binary-copy\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.317475 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-systemd-units\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.317475 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-host-slash\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.317475 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-systemd\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-run-systemd\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87a8beac-18a0-4743-9368-e0eea3507ad8-systemd-units\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/310f9f3d-4268-4c5b-b046-afa2cebb3685-dbus\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/310f9f3d-4268-4c5b-b046-afa2cebb3685-dbus\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-cni-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-sys\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-tuned\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-sys\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.317893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7tvv\" (UniqueName: \"kubernetes.io/projected/b19e5fc5-c807-4afb-a848-7cc107eb74a6-kube-api-access-r7tvv\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:49.318411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.318411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-k8s-cni-cncf-io\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.318411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-etc-kubernetes\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.318411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.317986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-kubernetes\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.318411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysctl-conf\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.318411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.318411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-kubernetes\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.318711 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-host\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.318711 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-sysctl-conf\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.318711 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-device-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.318711 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:49.318711 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-device-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.318711 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-netns\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.318711 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-host\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.319279 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.318862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.319279 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.318879    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:49.319279 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.318974    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs podName:b19e5fc5-c807-4afb-a848-7cc107eb74a6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:49.818952804 +0000 UTC m=+2.949499510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs") pod "network-metrics-daemon-wdr9l" (UID: "b19e5fc5-c807-4afb-a848-7cc107eb74a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:49.319636 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.319601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-socket-dir\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.320837 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.320797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af08d1a0-b107-4438-b7ed-8061c969ba8d-agent-certs\") pod \"konnectivity-agent-8tlk8\" (UID: \"af08d1a0-b107-4438-b7ed-8061c969ba8d\") " pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:40:49.321383 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.321354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-etc-tuned\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.321564 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.321536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-tmp\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.323235 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.323210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87a8beac-18a0-4743-9368-e0eea3507ad8-ovn-node-metrics-cert\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.324184 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.324163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q84bp\" (UniqueName: \"kubernetes.io/projected/68880f21-301d-4f7f-bb64-1f210f499111-kube-api-access-q84bp\") pod \"node-ca-q2cpl\" (UID: \"68880f21-301d-4f7f-bb64-1f210f499111\") " pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.326179 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.326153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d82j\" (UniqueName: \"kubernetes.io/projected/8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2-kube-api-access-4d82j\") pod \"tuned-9csgf\" (UID: \"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2\") " pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.331415 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.331366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal" event={"ID":"7b899a3c315c0814e264cff323a4907d","Type":"ContainerStarted","Data":"41bf8e8fab17175310cc8bf35da25c7fc61631fa828213560c02b32632bcb90f"}
Sep 16 18:40:49.332965 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.332917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal" event={"ID":"3ee3faaf407844b52f4750d947306742","Type":"ContainerStarted","Data":"1d31ab2d77dea5056d0b21c87c69007ce60a389c2f340715677d06dae2cc1c45"}
Sep 16 18:40:49.335235 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.335213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhhr\" (UniqueName: \"kubernetes.io/projected/e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181-kube-api-access-rwhhr\") pod \"aws-ebs-csi-driver-node-fwr8w\" (UID: \"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.338970 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.338944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7tvv\" (UniqueName: \"kubernetes.io/projected/b19e5fc5-c807-4afb-a848-7cc107eb74a6-kube-api-access-r7tvv\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:49.339502 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.339465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4n7j\" (UniqueName: \"kubernetes.io/projected/87a8beac-18a0-4743-9368-e0eea3507ad8-kube-api-access-s4n7j\") pod \"ovnkube-node-6jg58\" (UID: \"87a8beac-18a0-4743-9368-e0eea3507ad8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.367800 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.367767    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gl2l4"
Sep 16 18:40:49.380767 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.380739    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gl2l4"
Sep 16 18:40:49.419375 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-socket-dir-parent\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.419538 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.419538 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gn9v\" (UniqueName: \"kubernetes.io/projected/b708bf47-0a07-4e28-b464-487766a7377c-kube-api-access-5gn9v\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.419538 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-cni-bin\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.419538 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b2x8\" (UniqueName: \"kubernetes.io/projected/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-kube-api-access-8b2x8\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.419538 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-host-slash\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.419538 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b708bf47-0a07-4e28-b464-487766a7377c-hosts-file\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.419538 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-socket-dir-parent\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.419838 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-cni-bin\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420017 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-host-slash\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.420079 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b708bf47-0a07-4e28-b464-487766a7377c-hosts-file\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.420079 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.419524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-system-cni-dir\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420166 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2m4j\" (UniqueName: \"kubernetes.io/projected/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-kube-api-access-b2m4j\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.420166 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420166 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420310 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrls\" (UniqueName: \"kubernetes.io/projected/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-kube-api-access-8vrls\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420310 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-hostroot\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420310 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-multus-certs\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420310 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420310 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cnibin\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-cni-binary-copy\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-cni-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-k8s-cni-cncf-io\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cnibin\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-etc-kubernetes\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-system-cni-dir\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-hostroot\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-netns\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-multus-certs\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-cni-multus\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-conf-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-system-cni-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b708bf47-0a07-4e28-b464-487766a7377c-tmp-dir\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-cnibin\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-kubelet\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-daemon-config\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.420830 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-iptables-alerter-script\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.421226 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.420959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.421294 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-conf-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.421333 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-k8s-cni-cncf-io\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.421368 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-etc-kubernetes\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.421466 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-iptables-alerter-script\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.421550 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-system-cni-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.421550 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-run-netns\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.421627 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-cni-multus\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.421710 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-cni-binary-copy\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.421834 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b708bf47-0a07-4e28-b464-487766a7377c-tmp-dir\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.421928 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-cnibin\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.421982 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.421968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-host-var-lib-kubelet\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.422135 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.422115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.422224 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.422199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-cni-dir\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.422413 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.422387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-multus-daemon-config\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.428946 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.428622    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:49.428946 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.428651    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:49.428946 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.428665    2578 projected.go:196] Error preparing data for projected volume kube-api-access-z2tg7 for pod openshift-network-diagnostics/network-check-target-pgrc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:49.428946 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.428730    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7 podName:a15641d9-2640-49a8-a895-7b526b3a111e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:49.92871141 +0000 UTC m=+3.059258118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z2tg7" (UniqueName: "kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7") pod "network-check-target-pgrc2" (UID: "a15641d9-2640-49a8-a895-7b526b3a111e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:49.431558 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.431535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gn9v\" (UniqueName: \"kubernetes.io/projected/b708bf47-0a07-4e28-b464-487766a7377c-kube-api-access-5gn9v\") pod \"node-resolver-drqxf\" (UID: \"b708bf47-0a07-4e28-b464-487766a7377c\") " pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.431960 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.431938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b2x8\" (UniqueName: \"kubernetes.io/projected/f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5-kube-api-access-8b2x8\") pod \"multus-t8sp4\" (UID: \"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5\") " pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.432041 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.431939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrls\" (UniqueName: \"kubernetes.io/projected/2fdcb25e-2aad-4ca5-bca2-120ea66a478c-kube-api-access-8vrls\") pod \"multus-additional-cni-plugins-xx92x\" (UID: \"2fdcb25e-2aad-4ca5-bca2-120ea66a478c\") " pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.432105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.432038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2m4j\" (UniqueName: \"kubernetes.io/projected/e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4-kube-api-access-b2m4j\") pod \"iptables-alerter-w88db\" (UID: \"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4\") " pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.502979 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.502947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9csgf"
Sep 16 18:40:49.514295 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.514046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:40:49.524412 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.524390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:40:49.530359 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.530337    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w"
Sep 16 18:40:49.538883 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.538857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2cpl"
Sep 16 18:40:49.547601 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.547576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t8sp4"
Sep 16 18:40:49.557558 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.557542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w88db"
Sep 16 18:40:49.566247 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.566225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xx92x"
Sep 16 18:40:49.573139 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.573123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-drqxf"
Sep 16 18:40:49.668994 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.668958    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:49.824158 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.824111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:49.824401 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:49.824185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:49.824401 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.824312    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:49.824401 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.824380    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:49.824401 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.824394    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs podName:b19e5fc5-c807-4afb-a848-7cc107eb74a6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.824377174 +0000 UTC m=+3.954923879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs") pod "network-metrics-daemon-wdr9l" (UID: "b19e5fc5-c807-4afb-a848-7cc107eb74a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:49.824627 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:49.824447    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret podName:310f9f3d-4268-4c5b-b046-afa2cebb3685 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.824427143 +0000 UTC m=+3.954973852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret") pod "global-pull-secret-syncer-s25jm" (UID: "310f9f3d-4268-4c5b-b046-afa2cebb3685") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:50.026458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:50.026414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:50.026653 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.026584    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:50.026653 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.026607    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:50.026653 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.026622    2578 projected.go:196] Error preparing data for projected volume kube-api-access-z2tg7 for pod openshift-network-diagnostics/network-check-target-pgrc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:50.026787 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.026688    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7 podName:a15641d9-2640-49a8-a895-7b526b3a111e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:51.02667149 +0000 UTC m=+4.157218182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z2tg7" (UniqueName: "kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7") pod "network-check-target-pgrc2" (UID: "a15641d9-2640-49a8-a895-7b526b3a111e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:50.326371 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:50.326336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:50.326854 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.326492    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:40:50.384522 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:50.384478    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:50 +0000 UTC" deadline="2028-06-03 04:01:01.250301032 +0000 UTC"
Sep 16 18:40:50.384522 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:50.384518    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15009h20m10.865787535s"
Sep 16 18:40:50.415070 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:50.415046    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ecdc1f_c8c7_4541_92bc_1a2e78ed3181.slice/crio-7f5ae10e5bba10983a829d8ddab9d5b7e6788ce8c775cc29f1ded9f022faeaf1 WatchSource:0}: Error finding container 7f5ae10e5bba10983a829d8ddab9d5b7e6788ce8c775cc29f1ded9f022faeaf1: Status 404 returned error can't find the container with id 7f5ae10e5bba10983a829d8ddab9d5b7e6788ce8c775cc29f1ded9f022faeaf1
Sep 16 18:40:50.415590 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:50.415559    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8908c8a2_ccdc_4b9a_8d63_393dc3ec94f2.slice/crio-17d2509d65ff5bdac15be6605c8a7873f5ab2917a2e099453f08d0f0e5a02557 WatchSource:0}: Error finding container 17d2509d65ff5bdac15be6605c8a7873f5ab2917a2e099453f08d0f0e5a02557: Status 404 returned error can't find the container with id 17d2509d65ff5bdac15be6605c8a7873f5ab2917a2e099453f08d0f0e5a02557
Sep 16 18:40:50.418283 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:50.416828    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdcb25e_2aad_4ca5_bca2_120ea66a478c.slice/crio-2760ce890306c202f8cf986661cd96314adbad8236937a8af9dc56f0f559395c WatchSource:0}: Error finding container 2760ce890306c202f8cf986661cd96314adbad8236937a8af9dc56f0f559395c: Status 404 returned error can't find the container with id 2760ce890306c202f8cf986661cd96314adbad8236937a8af9dc56f0f559395c
Sep 16 18:40:50.422971 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:50.419171    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cd23c7_c03e_4b22_9fd4_ad6ea60c9ab4.slice/crio-5b0c5d78aec5a6a5fa20fe719e94e2480df3eb395c2ce7381411c0deb050db87 WatchSource:0}: Error finding container 5b0c5d78aec5a6a5fa20fe719e94e2480df3eb395c2ce7381411c0deb050db87: Status 404 returned error can't find the container with id 5b0c5d78aec5a6a5fa20fe719e94e2480df3eb395c2ce7381411c0deb050db87
Sep 16 18:40:50.422971 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:50.422370    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:50.425977 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:40:50.425800    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf08d1a0_b107_4438_b7ed_8061c969ba8d.slice/crio-ee2bfa76aa4624a07e27ea878362aa293b5bc1c16edb5e69349c144dba07df01 WatchSource:0}: Error finding container ee2bfa76aa4624a07e27ea878362aa293b5bc1c16edb5e69349c144dba07df01: Status 404 returned error can't find the container with id ee2bfa76aa4624a07e27ea878362aa293b5bc1c16edb5e69349c144dba07df01
Sep 16 18:40:50.832349 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:50.832306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:50.832524 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:50.832364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:50.832524 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.832513    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:50.832650 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.832581    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret podName:310f9f3d-4268-4c5b-b046-afa2cebb3685 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:52.832560767 +0000 UTC m=+5.963107477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret") pod "global-pull-secret-syncer-s25jm" (UID: "310f9f3d-4268-4c5b-b046-afa2cebb3685") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:50.833093 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.833043    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:50.833233 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:50.833126    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs podName:b19e5fc5-c807-4afb-a848-7cc107eb74a6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:52.833105301 +0000 UTC m=+5.963652008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs") pod "network-metrics-daemon-wdr9l" (UID: "b19e5fc5-c807-4afb-a848-7cc107eb74a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:51.034889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.034843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:51.035139 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:51.035061    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:51.035139 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:51.035089    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:51.035139 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:51.035103    2578 projected.go:196] Error preparing data for projected volume kube-api-access-z2tg7 for pod openshift-network-diagnostics/network-check-target-pgrc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:51.035342 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:51.035166    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7 podName:a15641d9-2640-49a8-a895-7b526b3a111e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:53.035146894 +0000 UTC m=+6.165693600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z2tg7" (UniqueName: "kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7") pod "network-check-target-pgrc2" (UID: "a15641d9-2640-49a8-a895-7b526b3a111e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:51.328709 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.327211    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:51.328709 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:51.327419    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:40:51.328709 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.327815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:51.328709 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:51.327920    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:40:51.351991 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.351950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w" event={"ID":"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181","Type":"ContainerStarted","Data":"7f5ae10e5bba10983a829d8ddab9d5b7e6788ce8c775cc29f1ded9f022faeaf1"}
Sep 16 18:40:51.355483 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.355453    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ee3faaf407844b52f4750d947306742" containerID="7888484d3efd5695400b6ae50448045232f4f3435b1db555ff933e6bafa3be78" exitCode=0
Sep 16 18:40:51.355627 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.355544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal" event={"ID":"3ee3faaf407844b52f4750d947306742","Type":"ContainerDied","Data":"7888484d3efd5695400b6ae50448045232f4f3435b1db555ff933e6bafa3be78"}
Sep 16 18:40:51.385428 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.385389    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:50 +0000 UTC" deadline="2028-05-08 03:07:38.574215798 +0000 UTC"
Sep 16 18:40:51.385428 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.385428    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14384h26m47.188791608s"
Sep 16 18:40:51.388421 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.388275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8tlk8" event={"ID":"af08d1a0-b107-4438-b7ed-8061c969ba8d","Type":"ContainerStarted","Data":"ee2bfa76aa4624a07e27ea878362aa293b5bc1c16edb5e69349c144dba07df01"}
Sep 16 18:40:51.405384 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.405326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w88db" event={"ID":"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4","Type":"ContainerStarted","Data":"5b0c5d78aec5a6a5fa20fe719e94e2480df3eb395c2ce7381411c0deb050db87"}
Sep 16 18:40:51.414683 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.414533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9csgf" event={"ID":"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2","Type":"ContainerStarted","Data":"17d2509d65ff5bdac15be6605c8a7873f5ab2917a2e099453f08d0f0e5a02557"}
Sep 16 18:40:51.421582 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.421546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal" event={"ID":"7b899a3c315c0814e264cff323a4907d","Type":"ContainerStarted","Data":"7cc0e456dadf5d2314f9efd2f14b7979220f4e10c639869f859ab0474dc8f440"}
Sep 16 18:40:51.434610 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.434577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-drqxf" event={"ID":"b708bf47-0a07-4e28-b464-487766a7377c","Type":"ContainerStarted","Data":"d427c9545208caf3916d723813bbad9e0f842589842dca041214b807854ffa9b"}
Sep 16 18:40:51.443259 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.443223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2cpl" event={"ID":"68880f21-301d-4f7f-bb64-1f210f499111","Type":"ContainerStarted","Data":"2d95cabfa00053c5c20cf08824cd053673fe4192aed8da02f0a7ede869035039"}
Sep 16 18:40:51.453348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.453316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t8sp4" event={"ID":"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5","Type":"ContainerStarted","Data":"50e3fefb50eab76e432af68be7e16736b305f777c0de2776ad415abc40676c7a"}
Sep 16 18:40:51.470952 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.470829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"5fa225b198f6f527597ef1abae4caa46b0c90887017f73e4748318eb5b48b655"}
Sep 16 18:40:51.495898 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:51.495795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx92x" event={"ID":"2fdcb25e-2aad-4ca5-bca2-120ea66a478c","Type":"ContainerStarted","Data":"2760ce890306c202f8cf986661cd96314adbad8236937a8af9dc56f0f559395c"}
Sep 16 18:40:52.326976 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:52.326409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:52.326976 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:52.326561    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:40:52.532255 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:52.532133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal" event={"ID":"3ee3faaf407844b52f4750d947306742","Type":"ContainerStarted","Data":"fdd45b81fe78ed6b7bce9e89a36da76d0f8ce78dcdbb38603507985333526b14"}
Sep 16 18:40:52.561955 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:52.561327    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal" podStartSLOduration=3.56130519 podStartE2EDuration="3.56130519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.442502728 +0000 UTC m=+4.573049446" watchObservedRunningTime="2026-09-16 18:40:52.56130519 +0000 UTC m=+5.691851906"
Sep 16 18:40:52.850772 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:52.850721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:52.850983 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:52.850828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:52.851097 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:52.851025    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:52.851162 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:52.851119    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret podName:310f9f3d-4268-4c5b-b046-afa2cebb3685 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:56.851098196 +0000 UTC m=+9.981644913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret") pod "global-pull-secret-syncer-s25jm" (UID: "310f9f3d-4268-4c5b-b046-afa2cebb3685") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:52.851236 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:52.851194    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:52.851236 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:52.851235    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs podName:b19e5fc5-c807-4afb-a848-7cc107eb74a6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:56.851221993 +0000 UTC m=+9.981768703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs") pod "network-metrics-daemon-wdr9l" (UID: "b19e5fc5-c807-4afb-a848-7cc107eb74a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:53.052465 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:53.052415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:53.052663 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:53.052643    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:53.052745 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:53.052672    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:53.052745 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:53.052687    2578 projected.go:196] Error preparing data for projected volume kube-api-access-z2tg7 for pod openshift-network-diagnostics/network-check-target-pgrc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:53.052848 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:53.052758    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7 podName:a15641d9-2640-49a8-a895-7b526b3a111e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:57.052736982 +0000 UTC m=+10.183283694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z2tg7" (UniqueName: "kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7") pod "network-check-target-pgrc2" (UID: "a15641d9-2640-49a8-a895-7b526b3a111e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:53.327702 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:53.327188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:53.327702 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:53.327198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:53.327702 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:53.327342    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:40:53.327702 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:53.327474    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:40:54.327135 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:54.327101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:54.327642 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:54.327243    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:40:55.327742 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:55.327176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:55.327742 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:55.327326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:40:55.328547 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:55.328384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:55.328547 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:55.328505    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:40:56.327309 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:56.327215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:56.327481 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:56.327363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:40:56.888764 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:56.888444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:56.888764 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:56.888558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:56.888764 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:56.888626    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:56.888764 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:56.888683    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:56.888764 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:56.888708    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret podName:310f9f3d-4268-4c5b-b046-afa2cebb3685 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:04.888685219 +0000 UTC m=+18.019231928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret") pod "global-pull-secret-syncer-s25jm" (UID: "310f9f3d-4268-4c5b-b046-afa2cebb3685") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:56.888764 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:56.888737    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs podName:b19e5fc5-c807-4afb-a848-7cc107eb74a6 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:04.888723123 +0000 UTC m=+18.019269814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs") pod "network-metrics-daemon-wdr9l" (UID: "b19e5fc5-c807-4afb-a848-7cc107eb74a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:57.091220 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:57.091168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:57.091421 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:57.091325    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:57.091421 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:57.091352    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:57.091421 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:57.091362    2578 projected.go:196] Error preparing data for projected volume kube-api-access-z2tg7 for pod openshift-network-diagnostics/network-check-target-pgrc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:57.091600 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:57.091434    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7 podName:a15641d9-2640-49a8-a895-7b526b3a111e nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.091413804 +0000 UTC m=+18.221960506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z2tg7" (UniqueName: "kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7") pod "network-check-target-pgrc2" (UID: "a15641d9-2640-49a8-a895-7b526b3a111e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:57.329505 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:57.329192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:57.329699 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:57.329636    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:40:57.330140 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:57.330113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:57.330266 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:57.330240    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:40:58.327253 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:58.327215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:40:58.327675 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:58.327356    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:40:59.327035 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:59.326982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:40:59.327329 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:59.327155    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:40:59.327820 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:40:59.327551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:40:59.327820 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:40:59.327652    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:00.326886 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:00.326851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:00.327066 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:00.326982    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:01.327236 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:01.327206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:01.327599 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:01.327206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:01.327599 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:01.327318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:01.327599 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:01.327372    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:02.327002 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:02.326966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:02.327186 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:02.327127    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:03.328151 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:03.328116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:03.328605 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:03.328251    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:03.328605 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:03.328316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:03.328605 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:03.328443    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:04.326840 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:04.326802    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:04.327122 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:04.326931    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:04.949388 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:04.949342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:04.949899 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:04.949406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:04.949899 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:04.949521    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:04.949899 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:04.949527    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:04.949899 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:04.949610    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs podName:b19e5fc5-c807-4afb-a848-7cc107eb74a6 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:20.949592403 +0000 UTC m=+34.080139093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs") pod "network-metrics-daemon-wdr9l" (UID: "b19e5fc5-c807-4afb-a848-7cc107eb74a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:04.949899 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:04.949675    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret podName:310f9f3d-4268-4c5b-b046-afa2cebb3685 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:20.949649496 +0000 UTC m=+34.080196188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret") pod "global-pull-secret-syncer-s25jm" (UID: "310f9f3d-4268-4c5b-b046-afa2cebb3685") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:05.151285 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:05.151247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:05.151482 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:05.151456    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:41:05.151482 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:05.151484    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:41:05.151594 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:05.151497    2578 projected.go:196] Error preparing data for projected volume kube-api-access-z2tg7 for pod openshift-network-diagnostics/network-check-target-pgrc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:05.151594 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:05.151563    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7 podName:a15641d9-2640-49a8-a895-7b526b3a111e nodeName:}" failed. No retries permitted until 2026-09-16 18:41:21.151544135 +0000 UTC m=+34.282090829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z2tg7" (UniqueName: "kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7") pod "network-check-target-pgrc2" (UID: "a15641d9-2640-49a8-a895-7b526b3a111e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:05.326622 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:05.326592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:05.326783 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:05.326713    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:05.326783 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:05.326748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:05.326872 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:05.326796    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:06.326969 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:06.326934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:06.327421 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:06.327078    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:07.327806 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:07.327777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:07.328228 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:07.327866    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:07.328228 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:07.327954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:07.328228 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:07.328073    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:08.327027 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:08.326977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:08.327215 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:08.327106    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:09.326437 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.326407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:09.327114 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.326407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:09.327114 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:09.326536    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:09.327114 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:09.326647    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:09.572326 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.572294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8tlk8" event={"ID":"af08d1a0-b107-4438-b7ed-8061c969ba8d","Type":"ContainerStarted","Data":"107052af49e5763688d74bee7352d48e715c558135932ce76916ff3bb97ab2ca"}
Sep 16 18:41:09.573715 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.573692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9csgf" event={"ID":"8908c8a2-ccdc-4b9a-8d63-393dc3ec94f2","Type":"ContainerStarted","Data":"75e4f52a2aa619f8db1c138492e3548e5aff34b626e9f55004aba6eb271f9cd8"}
Sep 16 18:41:09.577355 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.577329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-drqxf" event={"ID":"b708bf47-0a07-4e28-b464-487766a7377c","Type":"ContainerStarted","Data":"dd998a9b1e17fc359677b8c4e55b24492eb49d4bdcf810a02b51e25752693cf2"}
Sep 16 18:41:09.578800 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.578778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2cpl" event={"ID":"68880f21-301d-4f7f-bb64-1f210f499111","Type":"ContainerStarted","Data":"e962ba7bb397eb0c08576ec3a80690bd918c0e0cd0cbed00580f0263316469b7"}
Sep 16 18:41:09.580302 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.580276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t8sp4" event={"ID":"f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5","Type":"ContainerStarted","Data":"1a1872c28db6f259f7f71d99273ed349994babc8c4790ea32e3cf79584f24a2a"}
Sep 16 18:41:09.582790 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.582770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"3cc25b50261abc195d40f63c437318b89cfe381f24a0860bcfdeb760a2536d85"}
Sep 16 18:41:09.582877 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.582820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"3d3ba227bff2a7a286ae0b202e69c87110ab3ccd6a09fa3c52d4324079780503"}
Sep 16 18:41:09.582877 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.582855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"a443df16d98abe0e1227f49d6445b1aa4e2a42bbb71730156406df40a6e032aa"}
Sep 16 18:41:09.582958 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.582895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"47e36f67e1160f0cfc27b856ba5ddddce4be75b52f6668c442c5cbbbb8e3217d"}
Sep 16 18:41:09.582958 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.582927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"77a176333b8f787904dbf3825f6441a69ec61c8285ce24081b2463aa0a0373aa"}
Sep 16 18:41:09.584303 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.584284    2578 generic.go:356] "Generic (PLEG): container finished" podID="2fdcb25e-2aad-4ca5-bca2-120ea66a478c" containerID="348438acbe701c90fe648da167df0f071dbc797758f7e3687841145675e477c0" exitCode=0
Sep 16 18:41:09.584386 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.584344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx92x" event={"ID":"2fdcb25e-2aad-4ca5-bca2-120ea66a478c","Type":"ContainerDied","Data":"348438acbe701c90fe648da167df0f071dbc797758f7e3687841145675e477c0"}
Sep 16 18:41:09.585832 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.585814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w" event={"ID":"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181","Type":"ContainerStarted","Data":"1a86b185a156d6e54157727f6abf39bba360123ed5df7af60f820f56daf4c9f9"}
Sep 16 18:41:09.602587 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.602552    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal" podStartSLOduration=20.602539655 podStartE2EDuration="20.602539655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:52.561927661 +0000 UTC m=+5.692474376" watchObservedRunningTime="2026-09-16 18:41:09.602539655 +0000 UTC m=+22.733086365"
Sep 16 18:41:09.602886 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.602864    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8tlk8" podStartSLOduration=6.73798949 podStartE2EDuration="21.602858207s" totalImagesPullingTime="14.864868717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:09.602346611 +0000 UTC m=+22.732893321" watchObservedRunningTime="2026-09-16 18:41:09.602858207 +0000 UTC m=+22.733404917"
Sep 16 18:41:09.622806 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.622768    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9csgf" podStartSLOduration=2.986156743 podStartE2EDuration="21.622755309s" totalImagesPullingTime="18.636598566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:09.622128583 +0000 UTC m=+22.752675294" watchObservedRunningTime="2026-09-16 18:41:09.622755309 +0000 UTC m=+22.753302047"
Sep 16 18:41:09.664816 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.664760    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q2cpl" podStartSLOduration=3.085400838 podStartE2EDuration="21.664744211s" totalImagesPullingTime="18.579343373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:09.638526906 +0000 UTC m=+22.769073627" watchObservedRunningTime="2026-09-16 18:41:09.664744211 +0000 UTC m=+22.795290916"
Sep 16 18:41:09.665136 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.665097    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t8sp4" podStartSLOduration=3.009658125 podStartE2EDuration="21.66508723s" totalImagesPullingTime="18.655429105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:09.664916924 +0000 UTC m=+22.795463635" watchObservedRunningTime="2026-09-16 18:41:09.66508723 +0000 UTC m=+22.795633940"
Sep 16 18:41:09.728296 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:09.728242    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-drqxf" podStartSLOduration=3.140490119 podStartE2EDuration="21.728223484s" totalImagesPullingTime="18.587733365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:09.727952423 +0000 UTC m=+22.858499136" watchObservedRunningTime="2026-09-16 18:41:09.728223484 +0000 UTC m=+22.858770197"
Sep 16 18:41:10.326417 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:10.326353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:10.326541 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:10.326463    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:10.531620 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:10.531417    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:41:10.589964 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:10.589849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w88db" event={"ID":"e0cd23c7-c03e-4b22-9fd4-ad6ea60c9ab4","Type":"ContainerStarted","Data":"fbfe4bdfd7d3001cb9c7171f1946110fe15275a2d1524f6aafe5e1323a85d103"}
Sep 16 18:41:10.593542 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:10.593513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"6027d0e1206d8d9c7d48270df7f00e2e92a19892ca7ff47e69a3c085bf958245"}
Sep 16 18:41:10.596076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:10.596043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w" event={"ID":"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181","Type":"ContainerStarted","Data":"b603b6dad856dd60ab37f1467e4336e50ba4cd5e9293985929fa73f2adc399d0"}
Sep 16 18:41:10.607626 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:10.607579    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w88db" podStartSLOduration=4.013337581 podStartE2EDuration="22.607565794s" totalImagesPullingTime="18.594228213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:10.60711126 +0000 UTC m=+23.737657988" watchObservedRunningTime="2026-09-16 18:41:10.607565794 +0000 UTC m=+23.738112504"
Sep 16 18:41:11.279219 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:11.279091    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:41:10.53161726Z","UUID":"d1ad5b83-7f02-423e-b593-5684a9add583","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:41:11.280929 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:11.280896    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:41:11.280929 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:11.280934    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:41:11.326851 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:11.326825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:11.327291 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:11.326953    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:11.327365 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:11.326832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:11.327422 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:11.327401    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:11.600895 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:11.600848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w" event={"ID":"e6ecdc1f-c8c7-4541-92bc-1a2e78ed3181","Type":"ContainerStarted","Data":"d8c944fe9caa10b7c86e82c453c53265281d1109dfbdcbbdabd7ae10d825d8a0"}
Sep 16 18:41:11.625419 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:11.625343    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwr8w" podStartSLOduration=3.004347686 podStartE2EDuration="23.625327443s" totalImagesPullingTime="20.620979757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:11.625163852 +0000 UTC m=+24.755710616" watchObservedRunningTime="2026-09-16 18:41:11.625327443 +0000 UTC m=+24.755874151"
Sep 16 18:41:12.326227 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:12.326186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:12.326396 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:12.326304    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:12.607816 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:12.607735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"34c774674349137f93e1c851032164c96146cbd77bcbcba851d14950efc98ea2"}
Sep 16 18:41:13.326437 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:13.326399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:13.326616 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:13.326406    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:13.326616 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:13.326553    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:13.326704 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:13.326638    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:13.447396 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:13.447359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:41:13.448066 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:13.448046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:41:13.610436 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:13.610350    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:41:13.610915 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:13.610896    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8tlk8"
Sep 16 18:41:14.326508 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:14.326486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:14.326614 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:14.326595    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:14.616223 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:14.616191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" event={"ID":"87a8beac-18a0-4743-9368-e0eea3507ad8","Type":"ContainerStarted","Data":"eb34b3a99c7d0b51c4cdab4d59a472815197b67bc8022fe357953f6eb9e5ff37"}
Sep 16 18:41:14.616724 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:14.616533    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:41:14.616724 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:14.616590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:41:14.618355 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:14.618325    2578 generic.go:356] "Generic (PLEG): container finished" podID="2fdcb25e-2aad-4ca5-bca2-120ea66a478c" containerID="e6db71c38a8603e6a674b6b3cd7f1da5edf43e41adb32982746c6268a9554f2b" exitCode=0
Sep 16 18:41:14.618479 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:14.618419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx92x" event={"ID":"2fdcb25e-2aad-4ca5-bca2-120ea66a478c","Type":"ContainerDied","Data":"e6db71c38a8603e6a674b6b3cd7f1da5edf43e41adb32982746c6268a9554f2b"}
Sep 16 18:41:14.632277 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:14.632252    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:41:14.649661 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:14.649606    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58" podStartSLOduration=7.993571567 podStartE2EDuration="26.649588879s" totalImagesPullingTime="18.656017312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:14.649475205 +0000 UTC m=+27.780021946" watchObservedRunningTime="2026-09-16 18:41:14.649588879 +0000 UTC m=+27.780135609"
Sep 16 18:41:15.326710 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:15.326681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:15.326885 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:15.326785    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:15.326885 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:15.326813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:15.326885 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:15.326858    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:15.628134 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:15.628101    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:41:15.644915 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:15.644891    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:41:16.285603 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:16.285569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s25jm"]
Sep 16 18:41:16.285789 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:16.285713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:16.285860 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:16.285798    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:16.291661 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:16.291635    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wdr9l"]
Sep 16 18:41:16.291787 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:16.291745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:16.291877 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:16.291857    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:16.292147 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:16.292129    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pgrc2"]
Sep 16 18:41:16.292215 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:16.292204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:16.292289 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:16.292275    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:16.624894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:16.624825    2578 generic.go:356] "Generic (PLEG): container finished" podID="2fdcb25e-2aad-4ca5-bca2-120ea66a478c" containerID="ea6a0d5a6bee60eb6218971fcab309105b17a13fedf44936c23eb8ad6c0f6b86" exitCode=0
Sep 16 18:41:16.625041 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:16.624912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx92x" event={"ID":"2fdcb25e-2aad-4ca5-bca2-120ea66a478c","Type":"ContainerDied","Data":"ea6a0d5a6bee60eb6218971fcab309105b17a13fedf44936c23eb8ad6c0f6b86"}
Sep 16 18:41:18.327223 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:18.326993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:18.327682 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:18.327058    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:18.327682 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:18.327310    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:18.327682 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:18.327363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:18.327682 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:18.327070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:18.327682 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:18.327490    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:18.631743 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:18.631709    2578 generic.go:356] "Generic (PLEG): container finished" podID="2fdcb25e-2aad-4ca5-bca2-120ea66a478c" containerID="835384b98a80f9ce89068f855c463418c30ac9a0c079b0fa8179cd3051917feb" exitCode=0
Sep 16 18:41:18.631894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:18.631762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx92x" event={"ID":"2fdcb25e-2aad-4ca5-bca2-120ea66a478c","Type":"ContainerDied","Data":"835384b98a80f9ce89068f855c463418c30ac9a0c079b0fa8179cd3051917feb"}
Sep 16 18:41:20.326772 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:20.326708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:20.326772 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:20.326720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:20.327351 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:20.326830    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pgrc2" podUID="a15641d9-2640-49a8-a895-7b526b3a111e"
Sep 16 18:41:20.327351 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:20.327279    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdr9l" podUID="b19e5fc5-c807-4afb-a848-7cc107eb74a6"
Sep 16 18:41:20.327351 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:20.327332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:20.327511 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:20.327405    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s25jm" podUID="310f9f3d-4268-4c5b-b046-afa2cebb3685"
Sep 16 18:41:20.962944 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:20.962909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:20.963138 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:20.962960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:20.963138 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:20.963093    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:20.963248 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:20.963163    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs podName:b19e5fc5-c807-4afb-a848-7cc107eb74a6 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:52.963149403 +0000 UTC m=+66.093696106 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs") pod "network-metrics-daemon-wdr9l" (UID: "b19e5fc5-c807-4afb-a848-7cc107eb74a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:20.963248 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:20.963196    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:20.963338 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:20.963267    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret podName:310f9f3d-4268-4c5b-b046-afa2cebb3685 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:52.963248348 +0000 UTC m=+66.093795044 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret") pod "global-pull-secret-syncer-s25jm" (UID: "310f9f3d-4268-4c5b-b046-afa2cebb3685") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:21.164018 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.163970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:21.164172 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:21.164161    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:41:21.164216 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:21.164180    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:41:21.164216 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:21.164190    2578 projected.go:196] Error preparing data for projected volume kube-api-access-z2tg7 for pod openshift-network-diagnostics/network-check-target-pgrc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:21.164300 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:21.164243    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7 podName:a15641d9-2640-49a8-a895-7b526b3a111e nodeName:}" failed. No retries permitted until 2026-09-16 18:41:53.164230361 +0000 UTC m=+66.294777049 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z2tg7" (UniqueName: "kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7") pod "network-check-target-pgrc2" (UID: "a15641d9-2640-49a8-a895-7b526b3a111e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:21.620538 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.620509    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeReady"
Sep 16 18:41:21.620990 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.620677    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:41:21.687898 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.687867    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6rvpd"]
Sep 16 18:41:21.705708 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.705679    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h9vd7"]
Sep 16 18:41:21.705881 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.705830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.708924 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.708548    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:41:21.708924 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.708693    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:41:21.708924 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.708799    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zm88x\""
Sep 16 18:41:21.723995 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.723966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6rvpd"]
Sep 16 18:41:21.724097 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.724037    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h9vd7"]
Sep 16 18:41:21.724097 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.724079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h9vd7"
Sep 16 18:41:21.727059 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.727040    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:41:21.727178 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.727135    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:41:21.727323 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.727302    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2kfsb\""
Sep 16 18:41:21.727439 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.727363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:41:21.768596 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.768568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgzr\" (UniqueName: \"kubernetes.io/projected/4209b046-f8e0-49d9-b964-32593cd91d30-kube-api-access-9xgzr\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.768745 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.768653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.768745 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.768688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4209b046-f8e0-49d9-b964-32593cd91d30-tmp-dir\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.768745 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.768724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4209b046-f8e0-49d9-b964-32593cd91d30-config-volume\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.870026 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.869926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgzr\" (UniqueName: \"kubernetes.io/projected/4209b046-f8e0-49d9-b964-32593cd91d30-kube-api-access-9xgzr\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.870026 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.870000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.870317 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.870044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4209b046-f8e0-49d9-b964-32593cd91d30-tmp-dir\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.870317 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.870067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7089aa2c-4247-42fd-902e-83400da24150-cert\") pod \"ingress-canary-h9vd7\" (UID: \"7089aa2c-4247-42fd-902e-83400da24150\") " pod="openshift-ingress-canary/ingress-canary-h9vd7"
Sep 16 18:41:21.870317 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.870105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzkw\" (UniqueName: \"kubernetes.io/projected/7089aa2c-4247-42fd-902e-83400da24150-kube-api-access-jjzkw\") pod \"ingress-canary-h9vd7\" (UID: \"7089aa2c-4247-42fd-902e-83400da24150\") " pod="openshift-ingress-canary/ingress-canary-h9vd7"
Sep 16 18:41:21.870317 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:21.870171    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:21.870317 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:21.870250    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls podName:4209b046-f8e0-49d9-b964-32593cd91d30 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:22.370228529 +0000 UTC m=+35.500775241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls") pod "dns-default-6rvpd" (UID: "4209b046-f8e0-49d9-b964-32593cd91d30") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:21.870317 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.870170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4209b046-f8e0-49d9-b964-32593cd91d30-config-volume\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.870618 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.870400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4209b046-f8e0-49d9-b964-32593cd91d30-tmp-dir\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.870729 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.870706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4209b046-f8e0-49d9-b964-32593cd91d30-config-volume\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.888293 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.888267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgzr\" (UniqueName: \"kubernetes.io/projected/4209b046-f8e0-49d9-b964-32593cd91d30-kube-api-access-9xgzr\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:21.970975 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.970935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7089aa2c-4247-42fd-902e-83400da24150-cert\") pod \"ingress-canary-h9vd7\" (UID: \"7089aa2c-4247-42fd-902e-83400da24150\") " pod="openshift-ingress-canary/ingress-canary-h9vd7"
Sep 16 18:41:21.971216 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.970995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzkw\" (UniqueName: \"kubernetes.io/projected/7089aa2c-4247-42fd-902e-83400da24150-kube-api-access-jjzkw\") pod \"ingress-canary-h9vd7\" (UID: \"7089aa2c-4247-42fd-902e-83400da24150\") " pod="openshift-ingress-canary/ingress-canary-h9vd7"
Sep 16 18:41:21.974185 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.974152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7089aa2c-4247-42fd-902e-83400da24150-cert\") pod \"ingress-canary-h9vd7\" (UID: \"7089aa2c-4247-42fd-902e-83400da24150\") " pod="openshift-ingress-canary/ingress-canary-h9vd7"
Sep 16 18:41:21.985396 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:21.985373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzkw\" (UniqueName: \"kubernetes.io/projected/7089aa2c-4247-42fd-902e-83400da24150-kube-api-access-jjzkw\") pod \"ingress-canary-h9vd7\" (UID: \"7089aa2c-4247-42fd-902e-83400da24150\") " pod="openshift-ingress-canary/ingress-canary-h9vd7"
Sep 16 18:41:22.034408 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.034370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h9vd7"
Sep 16 18:41:22.327031 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.326976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:22.327294 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.326977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:22.327294 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.327130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:22.330722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.330495    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:22.330722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.330501    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:22.330722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.330570    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:22.330722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.330502    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx4q5\""
Sep 16 18:41:22.330722 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.330501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:22.331500 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.331480    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lk5bf\""
Sep 16 18:41:22.375045 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.375021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:22.375163 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:22.375106    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:22.375223 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:22.375179    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls podName:4209b046-f8e0-49d9-b964-32593cd91d30 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:23.375161913 +0000 UTC m=+36.505708602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls") pod "dns-default-6rvpd" (UID: "4209b046-f8e0-49d9-b964-32593cd91d30") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:22.976437 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:22.976259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h9vd7"]
Sep 16 18:41:22.989433 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:41:22.989394    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7089aa2c_4247_42fd_902e_83400da24150.slice/crio-91b55861c155eda41633833e49fd5489d784e3ff1cf26f7690bdb6595056e995 WatchSource:0}: Error finding container 91b55861c155eda41633833e49fd5489d784e3ff1cf26f7690bdb6595056e995: Status 404 returned error can't find the container with id 91b55861c155eda41633833e49fd5489d784e3ff1cf26f7690bdb6595056e995
Sep 16 18:41:23.383231 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:23.383201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:23.383383 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:23.383362    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:23.383456 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:23.383445    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls podName:4209b046-f8e0-49d9-b964-32593cd91d30 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:25.383428196 +0000 UTC m=+38.513974905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls") pod "dns-default-6rvpd" (UID: "4209b046-f8e0-49d9-b964-32593cd91d30") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:23.645233 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:23.645139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h9vd7" event={"ID":"7089aa2c-4247-42fd-902e-83400da24150","Type":"ContainerStarted","Data":"91b55861c155eda41633833e49fd5489d784e3ff1cf26f7690bdb6595056e995"}
Sep 16 18:41:23.648396 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:23.648361    2578 generic.go:356] "Generic (PLEG): container finished" podID="2fdcb25e-2aad-4ca5-bca2-120ea66a478c" containerID="f1059ecde711e3c333e5bae2973d50430db8b92658875fb41e0f2d3a3d4a693c" exitCode=0
Sep 16 18:41:23.648530 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:23.648415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx92x" event={"ID":"2fdcb25e-2aad-4ca5-bca2-120ea66a478c","Type":"ContainerDied","Data":"f1059ecde711e3c333e5bae2973d50430db8b92658875fb41e0f2d3a3d4a693c"}
Sep 16 18:41:24.654331 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:24.654293    2578 generic.go:356] "Generic (PLEG): container finished" podID="2fdcb25e-2aad-4ca5-bca2-120ea66a478c" containerID="a08ab478051b351ee64dc2e1f1a558ddf454a07601a44ac560aef4e547aa46cc" exitCode=0
Sep 16 18:41:24.654984 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:24.654343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx92x" event={"ID":"2fdcb25e-2aad-4ca5-bca2-120ea66a478c","Type":"ContainerDied","Data":"a08ab478051b351ee64dc2e1f1a558ddf454a07601a44ac560aef4e547aa46cc"}
Sep 16 18:41:25.404215 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:25.403820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:25.404215 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:25.403925    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:25.404215 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:25.404149    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls podName:4209b046-f8e0-49d9-b964-32593cd91d30 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:29.404129815 +0000 UTC m=+42.534676520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls") pod "dns-default-6rvpd" (UID: "4209b046-f8e0-49d9-b964-32593cd91d30") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:25.660064 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:25.659963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx92x" event={"ID":"2fdcb25e-2aad-4ca5-bca2-120ea66a478c","Type":"ContainerStarted","Data":"7ccf85380e7918a9c521ddedd90e1c433d82f45c2a2b18eeb1680cf7aafb21ee"}
Sep 16 18:41:25.661429 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:25.661408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h9vd7" event={"ID":"7089aa2c-4247-42fd-902e-83400da24150","Type":"ContainerStarted","Data":"10e12c686f8200ec7d0f9654317f80f2fb63e07f45b7eb2ff5f0f71fe61d5dba"}
Sep 16 18:41:25.689248 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:25.689189    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xx92x" podStartSLOduration=8.134939259 podStartE2EDuration="37.689145784s" totalImagesPullingTime="29.554206525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:25.687555067 +0000 UTC m=+38.818101792" watchObservedRunningTime="2026-09-16 18:41:25.689145784 +0000 UTC m=+38.819692495"
Sep 16 18:41:25.708912 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:25.708867    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h9vd7" podStartSLOduration=2.413980039 podStartE2EDuration="4.708850343s" totalImagesPullingTime="2.294870304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:25.708310941 +0000 UTC m=+38.838857652" watchObservedRunningTime="2026-09-16 18:41:25.708850343 +0000 UTC m=+38.839397081"
Sep 16 18:41:29.432927 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:29.432889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:29.433431 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:29.433090    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:29.433431 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:29.433174    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls podName:4209b046-f8e0-49d9-b964-32593cd91d30 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.433150938 +0000 UTC m=+50.563697641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls") pod "dns-default-6rvpd" (UID: "4209b046-f8e0-49d9-b964-32593cd91d30") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:37.486568 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:37.486517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:37.486933 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:37.486679    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:37.486933 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:37.486751    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls podName:4209b046-f8e0-49d9-b964-32593cd91d30 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:53.486731111 +0000 UTC m=+66.617277817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls") pod "dns-default-6rvpd" (UID: "4209b046-f8e0-49d9-b964-32593cd91d30") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:47.661448 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:47.661411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6jg58"
Sep 16 18:41:52.992219 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:52.992166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:41:52.992219 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:52.992226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:52.995488 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:52.995468    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:52.995535 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:52.995468    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:53.002693 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:53.002672    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:53.002768 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:53.002735    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs podName:b19e5fc5-c807-4afb-a848-7cc107eb74a6 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:57.002717582 +0000 UTC m=+130.133264270 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs") pod "network-metrics-daemon-wdr9l" (UID: "b19e5fc5-c807-4afb-a848-7cc107eb74a6") : secret "metrics-daemon-secret" not found
Sep 16 18:41:53.005942 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.005905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/310f9f3d-4268-4c5b-b046-afa2cebb3685-original-pull-secret\") pod \"global-pull-secret-syncer-s25jm\" (UID: \"310f9f3d-4268-4c5b-b046-afa2cebb3685\") " pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:53.194256 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.194218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:53.197367 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.197342    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:53.207412 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.207385    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:53.217255 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.217226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tg7\" (UniqueName: \"kubernetes.io/projected/a15641d9-2640-49a8-a895-7b526b3a111e-kube-api-access-z2tg7\") pod \"network-check-target-pgrc2\" (UID: \"a15641d9-2640-49a8-a895-7b526b3a111e\") " pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:53.240245 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.240223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s25jm"
Sep 16 18:41:53.257045 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.257027    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lk5bf\""
Sep 16 18:41:53.265246 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.265218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:53.496343 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.496274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:41:53.496477 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:53.496432    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:53.496515 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:41:53.496501    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls podName:4209b046-f8e0-49d9-b964-32593cd91d30 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:25.496482587 +0000 UTC m=+98.627029295 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls") pod "dns-default-6rvpd" (UID: "4209b046-f8e0-49d9-b964-32593cd91d30") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:53.518798 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.518773    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s25jm"]
Sep 16 18:41:53.522236 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:41:53.522212    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310f9f3d_4268_4c5b_b046_afa2cebb3685.slice/crio-e340ce627cbb6f1c10d5c6be49e4176e4dbfbcddea1411aa41988a0188135a32 WatchSource:0}: Error finding container e340ce627cbb6f1c10d5c6be49e4176e4dbfbcddea1411aa41988a0188135a32: Status 404 returned error can't find the container with id e340ce627cbb6f1c10d5c6be49e4176e4dbfbcddea1411aa41988a0188135a32
Sep 16 18:41:53.549143 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.549120    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pgrc2"]
Sep 16 18:41:53.552132 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:41:53.552113    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15641d9_2640_49a8_a895_7b526b3a111e.slice/crio-82feabb326ff2b903e153b06acce55ca416fda76ca047f9224e628f85a9b6a1a WatchSource:0}: Error finding container 82feabb326ff2b903e153b06acce55ca416fda76ca047f9224e628f85a9b6a1a: Status 404 returned error can't find the container with id 82feabb326ff2b903e153b06acce55ca416fda76ca047f9224e628f85a9b6a1a
Sep 16 18:41:53.725771 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.725737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pgrc2" event={"ID":"a15641d9-2640-49a8-a895-7b526b3a111e","Type":"ContainerStarted","Data":"82feabb326ff2b903e153b06acce55ca416fda76ca047f9224e628f85a9b6a1a"}
Sep 16 18:41:53.726858 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:53.726828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s25jm" event={"ID":"310f9f3d-4268-4c5b-b046-afa2cebb3685","Type":"ContainerStarted","Data":"e340ce627cbb6f1c10d5c6be49e4176e4dbfbcddea1411aa41988a0188135a32"}
Sep 16 18:41:58.743171 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:58.743128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pgrc2" event={"ID":"a15641d9-2640-49a8-a895-7b526b3a111e","Type":"ContainerStarted","Data":"0cdaee02af575c6ad390c99dca8d0101afb8362145a8df06818262bc308b6f54"}
Sep 16 18:41:58.743656 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:58.743251    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:41:58.744781 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:58.744750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s25jm" event={"ID":"310f9f3d-4268-4c5b-b046-afa2cebb3685","Type":"ContainerStarted","Data":"693a878c7c887f0dd178d6dd0a33d39732c7d801b790063ccc0d3cc93eecdbee"}
Sep 16 18:41:58.763595 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:58.763550    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pgrc2" podStartSLOduration=66.337777812 podStartE2EDuration="1m10.76353669s" totalImagesPullingTime="4.425758878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:58.762699052 +0000 UTC m=+71.893245761" watchObservedRunningTime="2026-09-16 18:41:58.76353669 +0000 UTC m=+71.894083439"
Sep 16 18:41:58.779487 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:41:58.779446    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-s25jm" podStartSLOduration=66.320552703 podStartE2EDuration="1m10.779433448s" totalImagesPullingTime="4.458880745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:58.779359421 +0000 UTC m=+71.909906144" watchObservedRunningTime="2026-09-16 18:41:58.779433448 +0000 UTC m=+71.909980191"
Sep 16 18:42:10.522831 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.522719    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-575r8"]
Sep 16 18:42:10.525156 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.525135    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57858ccb67-x4gf4"]
Sep 16 18:42:10.525277 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.525251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.527285 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.527264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.528588 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.528571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:42:10.528799 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.528782    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:42:10.529362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.529339    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:42:10.530157 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.530136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-57w29\""
Sep 16 18:42:10.530216 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.530171    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:42:10.530257 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.530173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:42:10.530876 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.530862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:42:10.531547 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.531530    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2ndk7\""
Sep 16 18:42:10.531547 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.531538    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:42:10.536603 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.536585    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:42:10.539782 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.539756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:42:10.543893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.543874    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-575r8"]
Sep 16 18:42:10.544964 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.544941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57858ccb67-x4gf4"]
Sep 16 18:42:10.604760 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7e40482-bb13-4380-8909-3dfc3a6fb77d-serving-cert\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.604760 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-bound-sa-token\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5pl\" (UniqueName: \"kubernetes.io/projected/e7e40482-bb13-4380-8909-3dfc3a6fb77d-kube-api-access-9c5pl\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d29ed02c-a64c-478a-b657-c91f6a34d371-ca-trust-extracted\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-installation-pull-secrets\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78l7v\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-kube-api-access-78l7v\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7e40482-bb13-4380-8909-3dfc3a6fb77d-trusted-ca\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-certificates\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7e40482-bb13-4380-8909-3dfc3a6fb77d-config\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.604997 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.604992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-trusted-ca\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.605446 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.605034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-image-registry-private-configuration\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.625314 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.625279    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"]
Sep 16 18:42:10.627499 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.627483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:10.630045 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.630001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:42:10.630045 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.630060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:42:10.630372 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.630357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-s28w6\""
Sep 16 18:42:10.641803 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.641774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"]
Sep 16 18:42:10.705592 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.705556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.705592 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.705593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-certificates\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.705592 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.705613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7e40482-bb13-4380-8909-3dfc3a6fb77d-config\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.705913 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.705631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-trusted-ca\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.705913 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.705657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-image-registry-private-configuration\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.705913 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.705696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:10.705913 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.705728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7e40482-bb13-4380-8909-3dfc3a6fb77d-serving-cert\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.705913 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:10.705732    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:10.705913 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:10.705754    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57858ccb67-x4gf4: secret "image-registry-tls" not found
Sep 16 18:42:10.705913 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:10.705819    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls podName:d29ed02c-a64c-478a-b657-c91f6a34d371 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:11.20579852 +0000 UTC m=+84.336345237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls") pod "image-registry-57858ccb67-x4gf4" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371") : secret "image-registry-tls" not found
Sep 16 18:42:10.706321 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.705974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:10.706321 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-bound-sa-token\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.706321 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5pl\" (UniqueName: \"kubernetes.io/projected/e7e40482-bb13-4380-8909-3dfc3a6fb77d-kube-api-access-9c5pl\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.706321 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d29ed02c-a64c-478a-b657-c91f6a34d371-ca-trust-extracted\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.706540 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-certificates\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.706540 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-installation-pull-secrets\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.706540 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78l7v\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-kube-api-access-78l7v\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.706540 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7e40482-bb13-4380-8909-3dfc3a6fb77d-trusted-ca\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.706743 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d29ed02c-a64c-478a-b657-c91f6a34d371-ca-trust-extracted\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.706743 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-trusted-ca\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.706743 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.706657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7e40482-bb13-4380-8909-3dfc3a6fb77d-config\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.707628 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.707598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7e40482-bb13-4380-8909-3dfc3a6fb77d-trusted-ca\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.708431 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.708407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7e40482-bb13-4380-8909-3dfc3a6fb77d-serving-cert\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.708851 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.708835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-image-registry-private-configuration\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.708981 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.708961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-installation-pull-secrets\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.723115 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.723088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5pl\" (UniqueName: \"kubernetes.io/projected/e7e40482-bb13-4380-8909-3dfc3a6fb77d-kube-api-access-9c5pl\") pod \"console-operator-696884fb9c-575r8\" (UID: \"e7e40482-bb13-4380-8909-3dfc3a6fb77d\") " pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:10.723367 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.723350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-bound-sa-token\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.724369 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.724350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78l7v\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-kube-api-access-78l7v\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:10.807429 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.807346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:10.807573 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.807455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:10.807573 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:10.807497    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:42:10.807653 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:10.807586    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert podName:e75dc3ff-9a16-4a71-bb1b-a7193a872c73 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:11.30756724 +0000 UTC m=+84.438113960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kx2rr" (UID: "e75dc3ff-9a16-4a71-bb1b-a7193a872c73") : secret "networking-console-plugin-cert" not found
Sep 16 18:42:10.808092 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.808076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:10.837061 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:10.837030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:11.108047 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:11.107937    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-575r8"]
Sep 16 18:42:11.111258 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:11.111233    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e40482_bb13_4380_8909_3dfc3a6fb77d.slice/crio-3f3674bce1e7191bdb3d0e3010881897b157836b609372c4c65ad9b96bae484b WatchSource:0}: Error finding container 3f3674bce1e7191bdb3d0e3010881897b157836b609372c4c65ad9b96bae484b: Status 404 returned error can't find the container with id 3f3674bce1e7191bdb3d0e3010881897b157836b609372c4c65ad9b96bae484b
Sep 16 18:42:11.210602 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:11.210564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:11.210767 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:11.210677    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:11.210767 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:11.210689    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57858ccb67-x4gf4: secret "image-registry-tls" not found
Sep 16 18:42:11.210767 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:11.210752    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls podName:d29ed02c-a64c-478a-b657-c91f6a34d371 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:12.210736593 +0000 UTC m=+85.341283282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls") pod "image-registry-57858ccb67-x4gf4" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371") : secret "image-registry-tls" not found
Sep 16 18:42:11.311266 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:11.311227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:11.311434 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:11.311339    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:42:11.311434 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:11.311395    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert podName:e75dc3ff-9a16-4a71-bb1b-a7193a872c73 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:12.311381281 +0000 UTC m=+85.441927983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kx2rr" (UID: "e75dc3ff-9a16-4a71-bb1b-a7193a872c73") : secret "networking-console-plugin-cert" not found
Sep 16 18:42:11.781023 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:11.780965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-575r8" event={"ID":"e7e40482-bb13-4380-8909-3dfc3a6fb77d","Type":"ContainerStarted","Data":"3f3674bce1e7191bdb3d0e3010881897b157836b609372c4c65ad9b96bae484b"}
Sep 16 18:42:12.219414 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:12.219323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:12.219587 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:12.219483    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:12.219587 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:12.219507    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57858ccb67-x4gf4: secret "image-registry-tls" not found
Sep 16 18:42:12.219587 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:12.219579    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls podName:d29ed02c-a64c-478a-b657-c91f6a34d371 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.219558772 +0000 UTC m=+87.350105475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls") pod "image-registry-57858ccb67-x4gf4" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371") : secret "image-registry-tls" not found
Sep 16 18:42:12.319876 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:12.319842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:12.320080 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:12.320036    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:42:12.320140 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:12.320106    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert podName:e75dc3ff-9a16-4a71-bb1b-a7193a872c73 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.320088033 +0000 UTC m=+87.450634735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kx2rr" (UID: "e75dc3ff-9a16-4a71-bb1b-a7193a872c73") : secret "networking-console-plugin-cert" not found
Sep 16 18:42:13.787914 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:13.787887    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/0.log"
Sep 16 18:42:13.788308 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:13.787926    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7e40482-bb13-4380-8909-3dfc3a6fb77d" containerID="d7a9e3e7d6b1324dea63c8e567b798f5fce80be17340297aa1c55eb04c1e942e" exitCode=255
Sep 16 18:42:13.788308 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:13.787960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-575r8" event={"ID":"e7e40482-bb13-4380-8909-3dfc3a6fb77d","Type":"ContainerDied","Data":"d7a9e3e7d6b1324dea63c8e567b798f5fce80be17340297aa1c55eb04c1e942e"}
Sep 16 18:42:13.788528 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:13.788508    2578 scope.go:117] "RemoveContainer" containerID="d7a9e3e7d6b1324dea63c8e567b798f5fce80be17340297aa1c55eb04c1e942e"
Sep 16 18:42:14.235502 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.235470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:14.235666 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.235646    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:14.235711 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.235669    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57858ccb67-x4gf4: secret "image-registry-tls" not found
Sep 16 18:42:14.235744 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.235732    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls podName:d29ed02c-a64c-478a-b657-c91f6a34d371 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.235715664 +0000 UTC m=+91.366262353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls") pod "image-registry-57858ccb67-x4gf4" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371") : secret "image-registry-tls" not found
Sep 16 18:42:14.336149 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.336102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:14.336320 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.336253    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:42:14.336320 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.336316    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert podName:e75dc3ff-9a16-4a71-bb1b-a7193a872c73 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.336298852 +0000 UTC m=+91.466845563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kx2rr" (UID: "e75dc3ff-9a16-4a71-bb1b-a7193a872c73") : secret "networking-console-plugin-cert" not found
Sep 16 18:42:14.610548 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.610469    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mzxqq"]
Sep 16 18:42:14.613385 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.613369    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.616823 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.616779    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:42:14.616823 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.616795    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:42:14.617024 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.616985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:42:14.617973 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.617954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-99dld\""
Sep 16 18:42:14.618101 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.617994    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:42:14.629996 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.629974    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mzxqq"]
Sep 16 18:42:14.638411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.638387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6863d054-55d0-4434-a64f-0d4b323b29a8-crio-socket\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.638524 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.638419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rjk\" (UniqueName: \"kubernetes.io/projected/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-api-access-n4rjk\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.638524 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.638438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.638524 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.638454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.638636 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.638531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6863d054-55d0-4434-a64f-0d4b323b29a8-data-volume\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.739085 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.739048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6863d054-55d0-4434-a64f-0d4b323b29a8-crio-socket\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.739283 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.739106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rjk\" (UniqueName: \"kubernetes.io/projected/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-api-access-n4rjk\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.739283 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.739139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.739283 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.739152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6863d054-55d0-4434-a64f-0d4b323b29a8-crio-socket\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.739283 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.739160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.739283 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.739235    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:14.739283 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.739250    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:14.739283 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.739285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6863d054-55d0-4434-a64f-0d4b323b29a8-data-volume\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.739586 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.739316    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.239296426 +0000 UTC m=+88.369843138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:14.739586 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.739383    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.239363074 +0000 UTC m=+88.369909789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:14.739665 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.739605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6863d054-55d0-4434-a64f-0d4b323b29a8-data-volume\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.749330 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.749299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rjk\" (UniqueName: \"kubernetes.io/projected/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-api-access-n4rjk\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:14.792487 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.792460    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:42:14.792944 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.792927    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/0.log"
Sep 16 18:42:14.792991 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.792964    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7e40482-bb13-4380-8909-3dfc3a6fb77d" containerID="d83ad971791309f3e387913f15c3e11ca1f7b682a5c2ea97521bcf567c5c1abd" exitCode=255
Sep 16 18:42:14.793058 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.793041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-575r8" event={"ID":"e7e40482-bb13-4380-8909-3dfc3a6fb77d","Type":"ContainerDied","Data":"d83ad971791309f3e387913f15c3e11ca1f7b682a5c2ea97521bcf567c5c1abd"}
Sep 16 18:42:14.793115 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.793103    2578 scope.go:117] "RemoveContainer" containerID="d7a9e3e7d6b1324dea63c8e567b798f5fce80be17340297aa1c55eb04c1e942e"
Sep 16 18:42:14.793271 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:14.793251    2578 scope.go:117] "RemoveContainer" containerID="d83ad971791309f3e387913f15c3e11ca1f7b682a5c2ea97521bcf567c5c1abd"
Sep 16 18:42:14.793433 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:14.793414    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-575r8_openshift-console-operator(e7e40482-bb13-4380-8909-3dfc3a6fb77d)\"" pod="openshift-console-operator/console-operator-696884fb9c-575r8" podUID="e7e40482-bb13-4380-8909-3dfc3a6fb77d"
Sep 16 18:42:15.243670 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.243631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:15.243670 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.243669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:15.243898 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:15.243755    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:15.243898 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:15.243819    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:16.243805837 +0000 UTC m=+89.374352551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:15.243898 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:15.243758    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:15.243898 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:15.243888    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:16.243877248 +0000 UTC m=+89.374423937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:15.568162 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.567991    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx"]
Sep 16 18:42:15.572165 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.572142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx"
Sep 16 18:42:15.575534 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.575510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q4v7w\""
Sep 16 18:42:15.576540 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.576523    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:42:15.576923 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.576910    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:42:15.587427 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.587404    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx"]
Sep 16 18:42:15.647401 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.647372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff78f\" (UniqueName: \"kubernetes.io/projected/6e8f7212-ea24-454f-af67-6961fd575bf4-kube-api-access-ff78f\") pod \"migrator-8b4d697c5-vrrxx\" (UID: \"6e8f7212-ea24-454f-af67-6961fd575bf4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx"
Sep 16 18:42:15.747783 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.747751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff78f\" (UniqueName: \"kubernetes.io/projected/6e8f7212-ea24-454f-af67-6961fd575bf4-kube-api-access-ff78f\") pod \"migrator-8b4d697c5-vrrxx\" (UID: \"6e8f7212-ea24-454f-af67-6961fd575bf4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx"
Sep 16 18:42:15.760289 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.760264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff78f\" (UniqueName: \"kubernetes.io/projected/6e8f7212-ea24-454f-af67-6961fd575bf4-kube-api-access-ff78f\") pod \"migrator-8b4d697c5-vrrxx\" (UID: \"6e8f7212-ea24-454f-af67-6961fd575bf4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx"
Sep 16 18:42:15.797811 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.797783    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:42:15.798280 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.798206    2578 scope.go:117] "RemoveContainer" containerID="d83ad971791309f3e387913f15c3e11ca1f7b682a5c2ea97521bcf567c5c1abd"
Sep 16 18:42:15.798403 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:15.798381    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-575r8_openshift-console-operator(e7e40482-bb13-4380-8909-3dfc3a6fb77d)\"" pod="openshift-console-operator/console-operator-696884fb9c-575r8" podUID="e7e40482-bb13-4380-8909-3dfc3a6fb77d"
Sep 16 18:42:15.880540 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:15.880431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx"
Sep 16 18:42:16.157808 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:16.157736    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx"]
Sep 16 18:42:16.160876 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:16.160846    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8f7212_ea24_454f_af67_6961fd575bf4.slice/crio-4416ccd47a9996eef0d70e5e4d941160ef0ce5c9efc31705ed183c522e6ae682 WatchSource:0}: Error finding container 4416ccd47a9996eef0d70e5e4d941160ef0ce5c9efc31705ed183c522e6ae682: Status 404 returned error can't find the container with id 4416ccd47a9996eef0d70e5e4d941160ef0ce5c9efc31705ed183c522e6ae682
Sep 16 18:42:16.251485 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:16.251452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:16.251620 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:16.251490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:16.251620 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:16.251573    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:16.251703 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:16.251635    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.251620067 +0000 UTC m=+91.382166757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:16.251703 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:16.251643    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:16.251703 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:16.251696    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.251683927 +0000 UTC m=+91.382230640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:16.310270 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:16.310242    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drqxf_b708bf47-0a07-4e28-b464-487766a7377c/dns-node-resolver/0.log"
Sep 16 18:42:16.802691 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:16.802648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx" event={"ID":"6e8f7212-ea24-454f-af67-6961fd575bf4","Type":"ContainerStarted","Data":"4416ccd47a9996eef0d70e5e4d941160ef0ce5c9efc31705ed183c522e6ae682"}
Sep 16 18:42:17.709526 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:17.709502    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2cpl_68880f21-301d-4f7f-bb64-1f210f499111/node-ca/0.log"
Sep 16 18:42:17.807462 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:17.807427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx" event={"ID":"6e8f7212-ea24-454f-af67-6961fd575bf4","Type":"ContainerStarted","Data":"5259cfa822c5054b0da1a27b952a0578534ec67fcca6db947811dc0b2ca4eb11"}
Sep 16 18:42:17.807910 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:17.807476    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx" event={"ID":"6e8f7212-ea24-454f-af67-6961fd575bf4","Type":"ContainerStarted","Data":"7e3a95605d340d7bdfc08604221ab699764a7e7205fc4e2c1ef97232e57b632e"}
Sep 16 18:42:18.269246 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:18.269217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:18.269246 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:18.269249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:18.269516 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:18.269272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:18.269516 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.269347    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.269516 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.269361    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.269516 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.269409    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:22.269393534 +0000 UTC m=+95.399940223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.269516 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.269423    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:22.269416856 +0000 UTC m=+95.399963544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.269516 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.269503    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:18.269750 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.269525    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57858ccb67-x4gf4: secret "image-registry-tls" not found
Sep 16 18:42:18.269750 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.269585    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls podName:d29ed02c-a64c-478a-b657-c91f6a34d371 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:26.269568009 +0000 UTC m=+99.400114713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls") pod "image-registry-57858ccb67-x4gf4" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371") : secret "image-registry-tls" not found
Sep 16 18:42:18.317423 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:18.317405    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h9vd7_7089aa2c-4247-42fd-902e-83400da24150/serve-healthcheck-canary/0.log"
Sep 16 18:42:18.370133 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:18.370104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:18.370233 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.370208    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:42:18.370267 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:18.370257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert podName:e75dc3ff-9a16-4a71-bb1b-a7193a872c73 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:26.370244488 +0000 UTC m=+99.500791177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kx2rr" (UID: "e75dc3ff-9a16-4a71-bb1b-a7193a872c73") : secret "networking-console-plugin-cert" not found
Sep 16 18:42:20.837208 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:20.837167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:20.837667 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:20.837219    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:20.837667 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:20.837527    2578 scope.go:117] "RemoveContainer" containerID="d83ad971791309f3e387913f15c3e11ca1f7b682a5c2ea97521bcf567c5c1abd"
Sep 16 18:42:20.837750 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:20.837671    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-575r8_openshift-console-operator(e7e40482-bb13-4380-8909-3dfc3a6fb77d)\"" pod="openshift-console-operator/console-operator-696884fb9c-575r8" podUID="e7e40482-bb13-4380-8909-3dfc3a6fb77d"
Sep 16 18:42:22.301280 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:22.301251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:22.301280 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:22.301285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:22.301688 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:22.301368    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:22.301688 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:22.301391    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:22.301688 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:22.301430    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:30.301415975 +0000 UTC m=+103.431962664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:22.301688 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:22.301506    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:30.301484348 +0000 UTC m=+103.432031043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:25.524863 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:25.524815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:42:25.527224 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:25.527195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4209b046-f8e0-49d9-b964-32593cd91d30-metrics-tls\") pod \"dns-default-6rvpd\" (UID: \"4209b046-f8e0-49d9-b964-32593cd91d30\") " pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:42:25.622518 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:25.622482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zm88x\""
Sep 16 18:42:25.630564 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:25.630543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:42:25.902228 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:25.902135    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vrrxx" podStartSLOduration=9.913133411 podStartE2EDuration="10.902116358s" totalImagesPullingTime="988.982947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.826432926 +0000 UTC m=+90.956979638" watchObservedRunningTime="2026-09-16 18:42:25.902116358 +0000 UTC m=+99.032663069"
Sep 16 18:42:25.903078 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:25.903048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6rvpd"]
Sep 16 18:42:25.905928 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:25.905901    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4209b046_f8e0_49d9_b964_32593cd91d30.slice/crio-45e3df332bab044dc3473470ed38251361772068237a062ac10513d25eea0cf5 WatchSource:0}: Error finding container 45e3df332bab044dc3473470ed38251361772068237a062ac10513d25eea0cf5: Status 404 returned error can't find the container with id 45e3df332bab044dc3473470ed38251361772068237a062ac10513d25eea0cf5
Sep 16 18:42:26.332788 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.332750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:26.335663 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.335641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"image-registry-57858ccb67-x4gf4\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") " pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:26.433515 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.433471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:26.433702 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:26.433639    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:42:26.433770 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:26.433730    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert podName:e75dc3ff-9a16-4a71-bb1b-a7193a872c73 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:42.433709373 +0000 UTC m=+115.564256079 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-kx2rr" (UID: "e75dc3ff-9a16-4a71-bb1b-a7193a872c73") : secret "networking-console-plugin-cert" not found
Sep 16 18:42:26.445320 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.445287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:26.735937 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.735904    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57858ccb67-x4gf4"]
Sep 16 18:42:26.739085 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:26.739047    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29ed02c_a64c_478a_b657_c91f6a34d371.slice/crio-4ea6aa2bb045183b44e7f90f496850c9873be46c8c3b036677cb81a7f8095cb8 WatchSource:0}: Error finding container 4ea6aa2bb045183b44e7f90f496850c9873be46c8c3b036677cb81a7f8095cb8: Status 404 returned error can't find the container with id 4ea6aa2bb045183b44e7f90f496850c9873be46c8c3b036677cb81a7f8095cb8
Sep 16 18:42:26.840865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.840831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4" event={"ID":"d29ed02c-a64c-478a-b657-c91f6a34d371","Type":"ContainerStarted","Data":"4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010"}
Sep 16 18:42:26.841088 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.840894    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:26.841088 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.840917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4" event={"ID":"d29ed02c-a64c-478a-b657-c91f6a34d371","Type":"ContainerStarted","Data":"4ea6aa2bb045183b44e7f90f496850c9873be46c8c3b036677cb81a7f8095cb8"}
Sep 16 18:42:26.842048 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.841997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rvpd" event={"ID":"4209b046-f8e0-49d9-b964-32593cd91d30","Type":"ContainerStarted","Data":"45e3df332bab044dc3473470ed38251361772068237a062ac10513d25eea0cf5"}
Sep 16 18:42:26.862094 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:26.862036    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4" podStartSLOduration=16.861993818 podStartE2EDuration="16.861993818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:26.8613107 +0000 UTC m=+99.991857453" watchObservedRunningTime="2026-09-16 18:42:26.861993818 +0000 UTC m=+99.992540528"
Sep 16 18:42:27.847256 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:27.847212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rvpd" event={"ID":"4209b046-f8e0-49d9-b964-32593cd91d30","Type":"ContainerStarted","Data":"afe8ecc2aea8d17605f20dbda58a7040edb58e95849391164babff7caccad912"}
Sep 16 18:42:27.847649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:27.847298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rvpd" event={"ID":"4209b046-f8e0-49d9-b964-32593cd91d30","Type":"ContainerStarted","Data":"2bed002f30f74bf61faaaf2b5e49c91f0795af2adff1e093b0b0a34d10250a33"}
Sep 16 18:42:27.847649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:27.847474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:42:27.869211 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:27.869166    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6rvpd" podStartSLOduration=65.440562206 podStartE2EDuration="1m6.869150901s" totalImagesPullingTime="1.428588695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:27.868137669 +0000 UTC m=+100.998684382" watchObservedRunningTime="2026-09-16 18:42:27.869150901 +0000 UTC m=+100.999697612"
Sep 16 18:42:29.749655 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:29.749628    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pgrc2"
Sep 16 18:42:30.365596 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:30.365558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:30.365814 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:30.365600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:30.365814 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:30.365767    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:30.365891 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:30.365837    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls podName:6863d054-55d0-4434-a64f-0d4b323b29a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:46.365818926 +0000 UTC m=+119.496365618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzxqq" (UID: "6863d054-55d0-4434-a64f-0d4b323b29a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:30.366238 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:30.366216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6863d054-55d0-4434-a64f-0d4b323b29a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:32.327127 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:32.327092    2578 scope.go:117] "RemoveContainer" containerID="d83ad971791309f3e387913f15c3e11ca1f7b682a5c2ea97521bcf567c5c1abd"
Sep 16 18:42:32.869166 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:32.869138    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:42:32.869319 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:32.869232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-575r8" event={"ID":"e7e40482-bb13-4380-8909-3dfc3a6fb77d","Type":"ContainerStarted","Data":"e04025434c34fa73f121444bcbb7e4c71e17300bb77bf53c679b42c4f693b00b"}
Sep 16 18:42:32.869535 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:32.869517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:32.876533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:32.876507    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-575r8"
Sep 16 18:42:32.890454 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:32.890400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-575r8" podStartSLOduration=20.799949235 podStartE2EDuration="22.890383311s" totalImagesPullingTime="2.090434076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:32.889686294 +0000 UTC m=+106.020233007" watchObservedRunningTime="2026-09-16 18:42:32.890383311 +0000 UTC m=+106.020930021"
Sep 16 18:42:37.852822 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:37.852792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6rvpd"
Sep 16 18:42:42.461216 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:42.461179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:42.464308 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:42.464286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e75dc3ff-9a16-4a71-bb1b-a7193a872c73-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kx2rr\" (UID: \"e75dc3ff-9a16-4a71-bb1b-a7193a872c73\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:42.735821 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:42.735792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"
Sep 16 18:42:43.010836 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.010756    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr"]
Sep 16 18:42:43.013756 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:43.013722    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75dc3ff_9a16_4a71_bb1b_a7193a872c73.slice/crio-1ee695fb3b60ac0db63f226315a2c5a48a9a697c53150c84b0d296b4d7c3a0a1 WatchSource:0}: Error finding container 1ee695fb3b60ac0db63f226315a2c5a48a9a697c53150c84b0d296b4d7c3a0a1: Status 404 returned error can't find the container with id 1ee695fb3b60ac0db63f226315a2c5a48a9a697c53150c84b0d296b4d7c3a0a1
Sep 16 18:42:43.613113 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.613083    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"]
Sep 16 18:42:43.615299 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.615277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.619208 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.619188    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:42:43.619310 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.619233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:42:43.619599 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.619582    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:42:43.620049 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.620021    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:42:43.620410 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.620386    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:42:43.620992 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.620823    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:42:43.620992 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.620819    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:42:43.628991 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.628972    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"]
Sep 16 18:42:43.632034 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.631991    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57858ccb67-x4gf4"]
Sep 16 18:42:43.637683 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.637634    2578 patch_prober.go:28] interesting pod/image-registry-57858ccb67-x4gf4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:42:43.637787 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.637690    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4" podUID="d29ed02c-a64c-478a-b657-c91f6a34d371" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:42:43.772663 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.772630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72n84\" (UniqueName: \"kubernetes.io/projected/8aa6d597-1e80-40e8-a84a-d2303e22884c-kube-api-access-72n84\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.772663 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.772666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-hub\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.772904 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.772699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-ca\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.772904 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.772812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.772904 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.772856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa6d597-1e80-40e8-a84a-d2303e22884c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.773049 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.772953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.873461 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.873375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72n84\" (UniqueName: \"kubernetes.io/projected/8aa6d597-1e80-40e8-a84a-d2303e22884c-kube-api-access-72n84\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.873461 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.873427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-hub\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.873680 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.873469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-ca\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.873680 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.873507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.873680 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.873534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa6d597-1e80-40e8-a84a-d2303e22884c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.873680 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.873599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.874546 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.874501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa6d597-1e80-40e8-a84a-d2303e22884c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.876517 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.876493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-ca\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.876774 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.876756    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.877117 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.877056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-hub\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.877344 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.877322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8aa6d597-1e80-40e8-a84a-d2303e22884c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.890974 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.889818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72n84\" (UniqueName: \"kubernetes.io/projected/8aa6d597-1e80-40e8-a84a-d2303e22884c-kube-api-access-72n84\") pod \"cluster-proxy-proxy-agent-7d69b9dfc-mvlrq\" (UID: \"8aa6d597-1e80-40e8-a84a-d2303e22884c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:43.908123 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.908091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr" event={"ID":"e75dc3ff-9a16-4a71-bb1b-a7193a872c73","Type":"ContainerStarted","Data":"1ee695fb3b60ac0db63f226315a2c5a48a9a697c53150c84b0d296b4d7c3a0a1"}
Sep 16 18:42:43.934149 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:43.934123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"
Sep 16 18:42:44.230921 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:44.230894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq"]
Sep 16 18:42:44.234030 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:44.233991    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8aa6d597_1e80_40e8_a84a_d2303e22884c.slice/crio-5ef9177c7d4804ad5b0cc2c93d1791d27f66210935c734027855e1f76e81f800 WatchSource:0}: Error finding container 5ef9177c7d4804ad5b0cc2c93d1791d27f66210935c734027855e1f76e81f800: Status 404 returned error can't find the container with id 5ef9177c7d4804ad5b0cc2c93d1791d27f66210935c734027855e1f76e81f800
Sep 16 18:42:44.915079 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:44.914985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr" event={"ID":"e75dc3ff-9a16-4a71-bb1b-a7193a872c73","Type":"ContainerStarted","Data":"4e1a0a0ae47e0d19978d681ff2600a24f9383abb49fe355f2fc6adb90bb60467"}
Sep 16 18:42:44.917223 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:44.916992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq" event={"ID":"8aa6d597-1e80-40e8-a84a-d2303e22884c","Type":"ContainerStarted","Data":"5ef9177c7d4804ad5b0cc2c93d1791d27f66210935c734027855e1f76e81f800"}
Sep 16 18:42:44.934968 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:44.934904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kx2rr" podStartSLOduration=33.963860334 podStartE2EDuration="34.934879836s" totalImagesPullingTime="971.019502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:44.933163696 +0000 UTC m=+118.063710409" watchObservedRunningTime="2026-09-16 18:42:44.934879836 +0000 UTC m=+118.065426549"
Sep 16 18:42:46.395684 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:46.395649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:46.398734 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:46.398701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6863d054-55d0-4434-a64f-0d4b323b29a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzxqq\" (UID: \"6863d054-55d0-4434-a64f-0d4b323b29a8\") " pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:46.422622 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:46.422583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mzxqq"
Sep 16 18:42:46.704810 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:46.704737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mzxqq"]
Sep 16 18:42:47.649565 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:47.649518    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6863d054_55d0_4434_a64f_0d4b323b29a8.slice/crio-8c55d04223aa14e027feeac6fb1a6f4b7aad32c6ec8235522eaf1b66c9e184ba WatchSource:0}: Error finding container 8c55d04223aa14e027feeac6fb1a6f4b7aad32c6ec8235522eaf1b66c9e184ba: Status 404 returned error can't find the container with id 8c55d04223aa14e027feeac6fb1a6f4b7aad32c6ec8235522eaf1b66c9e184ba
Sep 16 18:42:47.932524 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:47.932495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq" event={"ID":"8aa6d597-1e80-40e8-a84a-d2303e22884c","Type":"ContainerStarted","Data":"88fd89e41df0bce5c4728d1d04950108023380df42751dd13cc17580009b4f6a"}
Sep 16 18:42:47.932625 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:47.932544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq" event={"ID":"8aa6d597-1e80-40e8-a84a-d2303e22884c","Type":"ContainerStarted","Data":"f4e111c4382cc405f439faf120ac6a0ac41b64b71ff0a3f57aee860f609bd6ee"}
Sep 16 18:42:47.934342 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:47.934293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mzxqq" event={"ID":"6863d054-55d0-4434-a64f-0d4b323b29a8","Type":"ContainerStarted","Data":"acc8265ef9c43592b0f20bba926a332ffad8936aa473823b71a6f2cb5b0da323"}
Sep 16 18:42:47.934387 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:47.934355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mzxqq" event={"ID":"6863d054-55d0-4434-a64f-0d4b323b29a8","Type":"ContainerStarted","Data":"8c55d04223aa14e027feeac6fb1a6f4b7aad32c6ec8235522eaf1b66c9e184ba"}
Sep 16 18:42:48.939888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:48.939855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq" event={"ID":"8aa6d597-1e80-40e8-a84a-d2303e22884c","Type":"ContainerStarted","Data":"8d5d6b983a8b3d1842dd02bdd3d1310cb8d4ee49f385a29ad658a75ee8f6e33a"}
Sep 16 18:42:48.941689 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:48.941666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mzxqq" event={"ID":"6863d054-55d0-4434-a64f-0d4b323b29a8","Type":"ContainerStarted","Data":"62c239a1c6383539a7c02926cbca65308d3835adff902db7f6acb6ad9749b1c5"}
Sep 16 18:42:48.976246 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:48.976188    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d69b9dfc-mvlrq" podStartSLOduration=2.417061753 podStartE2EDuration="5.976167569s" totalImagesPullingTime="3.559105816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:48.97614352 +0000 UTC m=+122.106690233" watchObservedRunningTime="2026-09-16 18:42:48.976167569 +0000 UTC m=+122.106714271"
Sep 16 18:42:49.947293 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:49.947257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mzxqq" event={"ID":"6863d054-55d0-4434-a64f-0d4b323b29a8","Type":"ContainerStarted","Data":"435977c6dbf4263ffc3a859d337af71748a0030981eb6eb4b8efaf0015fdc9bf"}
Sep 16 18:42:49.969486 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:49.969438    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mzxqq" podStartSLOduration=33.98519448 podStartE2EDuration="35.969420844s" totalImagesPullingTime="1.984226364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:49.968316448 +0000 UTC m=+123.098863158" watchObservedRunningTime="2026-09-16 18:42:49.969420844 +0000 UTC m=+123.099967554"
Sep 16 18:42:53.636434 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:53.636407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:42:54.198960 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.198927    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wzh5b"]
Sep 16 18:42:54.202304 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.202286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.204785 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.204746    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:54.204916 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.204860    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:54.204916 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.204909    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:54.205184 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.205155    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5gflp\""
Sep 16 18:42:54.205184 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.205175    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:54.205314 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.205276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:54.206082 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.206067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:54.353370 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11420469-fbd0-4e40-91a1-8c649fd1765c-metrics-client-ca\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.353370 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.353576 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-sys\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.353576 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.353656 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-wtmp\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.353656 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8h9w\" (UniqueName: \"kubernetes.io/projected/11420469-fbd0-4e40-91a1-8c649fd1765c-kube-api-access-c8h9w\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.353656 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-textfile\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.353760 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.353760 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.353736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-root\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454470 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-sys\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454470 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454470 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-wtmp\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8h9w\" (UniqueName: \"kubernetes.io/projected/11420469-fbd0-4e40-91a1-8c649fd1765c-kube-api-access-c8h9w\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-sys\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-textfile\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-root\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-wtmp\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.454740 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11420469-fbd0-4e40-91a1-8c649fd1765c-metrics-client-ca\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.455124 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.455124 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11420469-fbd0-4e40-91a1-8c649fd1765c-root\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.455124 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:54.454880    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:42:54.455124 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.454887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-textfile\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.455124 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:54.454937    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls podName:11420469-fbd0-4e40-91a1-8c649fd1765c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:54.954920372 +0000 UTC m=+128.085467078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls") pod "node-exporter-wzh5b" (UID: "11420469-fbd0-4e40-91a1-8c649fd1765c") : secret "node-exporter-tls" not found
Sep 16 18:42:54.455124 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.455093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.455395 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.455276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11420469-fbd0-4e40-91a1-8c649fd1765c-metrics-client-ca\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.457112 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.457092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.463999 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.463980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8h9w\" (UniqueName: \"kubernetes.io/projected/11420469-fbd0-4e40-91a1-8c649fd1765c-kube-api-access-c8h9w\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.959739 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:54.959699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:54.960272 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:54.959849    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:42:54.960272 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:42:54.959942    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls podName:11420469-fbd0-4e40-91a1-8c649fd1765c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:55.959921794 +0000 UTC m=+129.090468484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls") pod "node-exporter-wzh5b" (UID: "11420469-fbd0-4e40-91a1-8c649fd1765c") : secret "node-exporter-tls" not found
Sep 16 18:42:55.968200 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:55.968171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:55.970589 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:55.970571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11420469-fbd0-4e40-91a1-8c649fd1765c-node-exporter-tls\") pod \"node-exporter-wzh5b\" (UID: \"11420469-fbd0-4e40-91a1-8c649fd1765c\") " pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:56.010717 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:56.010690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wzh5b"
Sep 16 18:42:56.021902 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:56.021877    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11420469_fbd0_4e40_91a1_8c649fd1765c.slice/crio-b4f4689cfd91793263403fa976a05e7f8a79a4d0532edb19d181bafa6693f17f WatchSource:0}: Error finding container b4f4689cfd91793263403fa976a05e7f8a79a4d0532edb19d181bafa6693f17f: Status 404 returned error can't find the container with id b4f4689cfd91793263403fa976a05e7f8a79a4d0532edb19d181bafa6693f17f
Sep 16 18:42:56.971987 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:56.971950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzh5b" event={"ID":"11420469-fbd0-4e40-91a1-8c649fd1765c","Type":"ContainerStarted","Data":"ce8274f239bd3cd02eadfefc8c4c8561855073395e16f58f070659cf084c2778"}
Sep 16 18:42:56.972428 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:56.972048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzh5b" event={"ID":"11420469-fbd0-4e40-91a1-8c649fd1765c","Type":"ContainerStarted","Data":"b4f4689cfd91793263403fa976a05e7f8a79a4d0532edb19d181bafa6693f17f"}
Sep 16 18:42:57.077314 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:57.077274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:42:57.079529 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:57.079501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b19e5fc5-c807-4afb-a848-7cc107eb74a6-metrics-certs\") pod \"network-metrics-daemon-wdr9l\" (UID: \"b19e5fc5-c807-4afb-a848-7cc107eb74a6\") " pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:42:57.150811 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:57.150740    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx4q5\""
Sep 16 18:42:57.158210 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:57.158189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdr9l"
Sep 16 18:42:57.431038 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:57.430949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wdr9l"]
Sep 16 18:42:57.433796 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:42:57.433760    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19e5fc5_c807_4afb_a848_7cc107eb74a6.slice/crio-4f3a0ada2369a26e098e57b19d86425a60df5bb95eef03470cf9e0a192c2350e WatchSource:0}: Error finding container 4f3a0ada2369a26e098e57b19d86425a60df5bb95eef03470cf9e0a192c2350e: Status 404 returned error can't find the container with id 4f3a0ada2369a26e098e57b19d86425a60df5bb95eef03470cf9e0a192c2350e
Sep 16 18:42:57.977852 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:57.977809    2578 generic.go:356] "Generic (PLEG): container finished" podID="11420469-fbd0-4e40-91a1-8c649fd1765c" containerID="ce8274f239bd3cd02eadfefc8c4c8561855073395e16f58f070659cf084c2778" exitCode=0
Sep 16 18:42:57.978308 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:57.977887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzh5b" event={"ID":"11420469-fbd0-4e40-91a1-8c649fd1765c","Type":"ContainerDied","Data":"ce8274f239bd3cd02eadfefc8c4c8561855073395e16f58f070659cf084c2778"}
Sep 16 18:42:57.979374 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:57.979350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdr9l" event={"ID":"b19e5fc5-c807-4afb-a848-7cc107eb74a6","Type":"ContainerStarted","Data":"4f3a0ada2369a26e098e57b19d86425a60df5bb95eef03470cf9e0a192c2350e"}
Sep 16 18:42:58.984770 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:58.984735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdr9l" event={"ID":"b19e5fc5-c807-4afb-a848-7cc107eb74a6","Type":"ContainerStarted","Data":"74501e97fcf962c68362042c67282763988297c1fcbb01039c56e88b487d254e"}
Sep 16 18:42:58.985259 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:58.984803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdr9l" event={"ID":"b19e5fc5-c807-4afb-a848-7cc107eb74a6","Type":"ContainerStarted","Data":"44056308e99aa016d299b026876bf779a85e68e0544b04ad2bf70c92aec6372b"}
Sep 16 18:42:58.986813 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:58.986793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzh5b" event={"ID":"11420469-fbd0-4e40-91a1-8c649fd1765c","Type":"ContainerStarted","Data":"4b9d8ad3d6f905d7a7495c9b97fd883f322eb8c258a055f1be30b6174c1f0959"}
Sep 16 18:42:58.986915 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:58.986836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzh5b" event={"ID":"11420469-fbd0-4e40-91a1-8c649fd1765c","Type":"ContainerStarted","Data":"6666d331d23712e02f1d75ef492d1e82e68996e9d0c7ab829be3dc8d6743df78"}
Sep 16 18:42:59.004923 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:59.004884    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wdr9l" podStartSLOduration=130.035489357 podStartE2EDuration="2m11.004869584s" totalImagesPullingTime="969.380227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:59.004113578 +0000 UTC m=+132.134660290" watchObservedRunningTime="2026-09-16 18:42:59.004869584 +0000 UTC m=+132.135416311"
Sep 16 18:42:59.029122 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:42:59.029068    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wzh5b" podStartSLOduration=4.096718241 podStartE2EDuration="5.029051338s" totalImagesPullingTime="863.075366ms" totalInitContainerRuntime="69.257731ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:59.029039494 +0000 UTC m=+132.159586208" watchObservedRunningTime="2026-09-16 18:42:59.029051338 +0000 UTC m=+132.159598046"
Sep 16 18:43:08.657452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:08.657381    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4" podUID="d29ed02c-a64c-478a-b657-c91f6a34d371" containerName="registry" containerID="cri-o://4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010" gracePeriod=30
Sep 16 18:43:08.887195 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:08.887173    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:43:09.019754 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.019718    2578 generic.go:356] "Generic (PLEG): container finished" podID="d29ed02c-a64c-478a-b657-c91f6a34d371" containerID="4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010" exitCode=0
Sep 16 18:43:09.019920 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.019766    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4"
Sep 16 18:43:09.019920 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.019812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4" event={"ID":"d29ed02c-a64c-478a-b657-c91f6a34d371","Type":"ContainerDied","Data":"4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010"}
Sep 16 18:43:09.019920 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.019864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57858ccb67-x4gf4" event={"ID":"d29ed02c-a64c-478a-b657-c91f6a34d371","Type":"ContainerDied","Data":"4ea6aa2bb045183b44e7f90f496850c9873be46c8c3b036677cb81a7f8095cb8"}
Sep 16 18:43:09.019920 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.019893    2578 scope.go:117] "RemoveContainer" containerID="4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010"
Sep 16 18:43:09.027598 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.027580    2578 scope.go:117] "RemoveContainer" containerID="4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010"
Sep 16 18:43:09.027874 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:43:09.027856    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010\": container with ID starting with 4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010 not found: ID does not exist" containerID="4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010"
Sep 16 18:43:09.027936 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.027885    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010"} err="failed to get container status \"4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010\": rpc error: code = NotFound desc = could not find container \"4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010\": container with ID starting with 4febbac83944b83028b0aed8155f5b6a26128f3af05f02a43c738a65a2800010 not found: ID does not exist"
Sep 16 18:43:09.069293 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-bound-sa-token\") pod \"d29ed02c-a64c-478a-b657-c91f6a34d371\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") "
Sep 16 18:43:09.069371 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069307    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d29ed02c-a64c-478a-b657-c91f6a34d371-ca-trust-extracted\") pod \"d29ed02c-a64c-478a-b657-c91f6a34d371\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") "
Sep 16 18:43:09.069371 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78l7v\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-kube-api-access-78l7v\") pod \"d29ed02c-a64c-478a-b657-c91f6a34d371\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") "
Sep 16 18:43:09.069433 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-trusted-ca\") pod \"d29ed02c-a64c-478a-b657-c91f6a34d371\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") "
Sep 16 18:43:09.069433 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069407    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-installation-pull-secrets\") pod \"d29ed02c-a64c-478a-b657-c91f6a34d371\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") "
Sep 16 18:43:09.069516 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-image-registry-private-configuration\") pod \"d29ed02c-a64c-478a-b657-c91f6a34d371\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") "
Sep 16 18:43:09.069516 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") pod \"d29ed02c-a64c-478a-b657-c91f6a34d371\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") "
Sep 16 18:43:09.069516 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-certificates\") pod \"d29ed02c-a64c-478a-b657-c91f6a34d371\" (UID: \"d29ed02c-a64c-478a-b657-c91f6a34d371\") "
Sep 16 18:43:09.069953 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.069826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d29ed02c-a64c-478a-b657-c91f6a34d371" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:09.070298 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.070241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d29ed02c-a64c-478a-b657-c91f6a34d371" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:09.071901 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.071877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d29ed02c-a64c-478a-b657-c91f6a34d371" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:09.072180 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.071920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d29ed02c-a64c-478a-b657-c91f6a34d371" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:09.072290 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.072266    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-kube-api-access-78l7v" (OuterVolumeSpecName: "kube-api-access-78l7v") pod "d29ed02c-a64c-478a-b657-c91f6a34d371" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371"). InnerVolumeSpecName "kube-api-access-78l7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:09.072350 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.072289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d29ed02c-a64c-478a-b657-c91f6a34d371" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:09.072350 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.072320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d29ed02c-a64c-478a-b657-c91f6a34d371" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:09.078310 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.078286    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29ed02c-a64c-478a-b657-c91f6a34d371-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d29ed02c-a64c-478a-b657-c91f6a34d371" (UID: "d29ed02c-a64c-478a-b657-c91f6a34d371"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:09.170085 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.170051    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-certificates\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:43:09.170085 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.170078    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-bound-sa-token\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:43:09.170085 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.170087    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d29ed02c-a64c-478a-b657-c91f6a34d371-ca-trust-extracted\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:43:09.170085 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.170096    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78l7v\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-kube-api-access-78l7v\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:43:09.170343 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.170105    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d29ed02c-a64c-478a-b657-c91f6a34d371-trusted-ca\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:43:09.170343 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.170113    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-installation-pull-secrets\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:43:09.170343 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.170122    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d29ed02c-a64c-478a-b657-c91f6a34d371-image-registry-private-configuration\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:43:09.170343 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.170131    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d29ed02c-a64c-478a-b657-c91f6a34d371-registry-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:43:09.342709 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.342679    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57858ccb67-x4gf4"]
Sep 16 18:43:09.352972 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:09.352949    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57858ccb67-x4gf4"]
Sep 16 18:43:11.330550 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:11.330516    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29ed02c-a64c-478a-b657-c91f6a34d371" path="/var/lib/kubelet/pods/d29ed02c-a64c-478a-b657-c91f6a34d371/volumes"
Sep 16 18:43:35.568496 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:43:35.568453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h9vd7_7089aa2c-4247-42fd-902e-83400da24150/serve-healthcheck-canary/0.log"
Sep 16 18:45:45.927671 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.927634    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"]
Sep 16 18:45:45.928085 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.927863    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29ed02c-a64c-478a-b657-c91f6a34d371" containerName="registry"
Sep 16 18:45:45.928085 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.927874    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29ed02c-a64c-478a-b657-c91f6a34d371" containerName="registry"
Sep 16 18:45:45.928085 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.927915    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29ed02c-a64c-478a-b657-c91f6a34d371" containerName="registry"
Sep 16 18:45:45.930756 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.930740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:45.933795 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.933778    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:45:45.933906 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.933864    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:45:45.934764 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.934749    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phwrs\""
Sep 16 18:45:45.939400 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.939377    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"]
Sep 16 18:45:45.949156 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.949131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:45.949270 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.949179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9d5f\" (UniqueName: \"kubernetes.io/projected/b1c0d68f-7af6-45a7-b733-5ea438a086e1-kube-api-access-r9d5f\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:45.949270 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:45.949205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:46.050152 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.050104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:46.050357 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.050172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9d5f\" (UniqueName: \"kubernetes.io/projected/b1c0d68f-7af6-45a7-b733-5ea438a086e1-kube-api-access-r9d5f\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:46.050357 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.050206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:46.050515 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.050495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:46.050583 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.050567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:46.061874 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.061844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9d5f\" (UniqueName: \"kubernetes.io/projected/b1c0d68f-7af6-45a7-b733-5ea438a086e1-kube-api-access-r9d5f\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:46.239592 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.239557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:45:46.510662 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.510582    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"]
Sep 16 18:45:46.513482 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:45:46.513456    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c0d68f_7af6_45a7_b733_5ea438a086e1.slice/crio-6604dc550c4565d2c674caedbbd6321d29a533e470504c525616850d13d6a1eb WatchSource:0}: Error finding container 6604dc550c4565d2c674caedbbd6321d29a533e470504c525616850d13d6a1eb: Status 404 returned error can't find the container with id 6604dc550c4565d2c674caedbbd6321d29a533e470504c525616850d13d6a1eb
Sep 16 18:45:46.534850 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:46.534814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9" event={"ID":"b1c0d68f-7af6-45a7-b733-5ea438a086e1","Type":"ContainerStarted","Data":"6604dc550c4565d2c674caedbbd6321d29a533e470504c525616850d13d6a1eb"}
Sep 16 18:45:47.206977 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:47.206939    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:45:47.208484 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:47.208460    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:45:47.219642 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:47.219618    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:52.562027 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:52.561980    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerID="503e36e3a047654b9784be456b29d22c677e400c78fbc448eca7cc633b15d993" exitCode=0
Sep 16 18:45:52.562027 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:52.562032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9" event={"ID":"b1c0d68f-7af6-45a7-b733-5ea438a086e1","Type":"ContainerDied","Data":"503e36e3a047654b9784be456b29d22c677e400c78fbc448eca7cc633b15d993"}
Sep 16 18:45:52.563106 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:52.563077    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:54.571076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:54.571047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9" event={"ID":"b1c0d68f-7af6-45a7-b733-5ea438a086e1","Type":"ContainerStarted","Data":"f10edbd59c25a84dd7326a9caa1dd6ac0556b89cc1d9630771102e6d1da17757"}
Sep 16 18:45:55.576697 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:55.576657    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerID="f10edbd59c25a84dd7326a9caa1dd6ac0556b89cc1d9630771102e6d1da17757" exitCode=0
Sep 16 18:45:55.577077 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:45:55.576726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9" event={"ID":"b1c0d68f-7af6-45a7-b733-5ea438a086e1","Type":"ContainerDied","Data":"f10edbd59c25a84dd7326a9caa1dd6ac0556b89cc1d9630771102e6d1da17757"}
Sep 16 18:46:01.602124 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:01.602091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9" event={"ID":"b1c0d68f-7af6-45a7-b733-5ea438a086e1","Type":"ContainerStarted","Data":"a3b64fa57c54a81d2360b9af6c14d3bf1f9b545d3452351c8c5c4c14c7ccaf01"}
Sep 16 18:46:01.622330 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:01.622282    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9" podStartSLOduration=3.645718339 podStartE2EDuration="16.622265495s" totalImagesPullingTime="12.976547156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:01.62155214 +0000 UTC m=+314.752098851" watchObservedRunningTime="2026-09-16 18:46:01.622265495 +0000 UTC m=+314.752812203"
Sep 16 18:46:02.608077 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:02.608037    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerID="a3b64fa57c54a81d2360b9af6c14d3bf1f9b545d3452351c8c5c4c14c7ccaf01" exitCode=0
Sep 16 18:46:02.608439 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:02.608082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9" event={"ID":"b1c0d68f-7af6-45a7-b733-5ea438a086e1","Type":"ContainerDied","Data":"a3b64fa57c54a81d2360b9af6c14d3bf1f9b545d3452351c8c5c4c14c7ccaf01"}
Sep 16 18:46:03.727649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.727623    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:46:03.777510 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.777485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-bundle\") pod \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") "
Sep 16 18:46:03.777623 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.777524    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9d5f\" (UniqueName: \"kubernetes.io/projected/b1c0d68f-7af6-45a7-b733-5ea438a086e1-kube-api-access-r9d5f\") pod \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") "
Sep 16 18:46:03.777623 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.777607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-util\") pod \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\" (UID: \"b1c0d68f-7af6-45a7-b733-5ea438a086e1\") "
Sep 16 18:46:03.778181 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.778155    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-bundle" (OuterVolumeSpecName: "bundle") pod "b1c0d68f-7af6-45a7-b733-5ea438a086e1" (UID: "b1c0d68f-7af6-45a7-b733-5ea438a086e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:03.779791 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.779765    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c0d68f-7af6-45a7-b733-5ea438a086e1-kube-api-access-r9d5f" (OuterVolumeSpecName: "kube-api-access-r9d5f") pod "b1c0d68f-7af6-45a7-b733-5ea438a086e1" (UID: "b1c0d68f-7af6-45a7-b733-5ea438a086e1"). InnerVolumeSpecName "kube-api-access-r9d5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:03.783039 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.782999    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-util" (OuterVolumeSpecName: "util") pod "b1c0d68f-7af6-45a7-b733-5ea438a086e1" (UID: "b1c0d68f-7af6-45a7-b733-5ea438a086e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:03.878120 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.878053    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-bundle\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:46:03.878120 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.878076    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9d5f\" (UniqueName: \"kubernetes.io/projected/b1c0d68f-7af6-45a7-b733-5ea438a086e1-kube-api-access-r9d5f\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:46:03.878120 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:03.878087    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c0d68f-7af6-45a7-b733-5ea438a086e1-util\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:46:04.616257 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:04.616221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9" event={"ID":"b1c0d68f-7af6-45a7-b733-5ea438a086e1","Type":"ContainerDied","Data":"6604dc550c4565d2c674caedbbd6321d29a533e470504c525616850d13d6a1eb"}
Sep 16 18:46:04.616415 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:04.616267    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6604dc550c4565d2c674caedbbd6321d29a533e470504c525616850d13d6a1eb"
Sep 16 18:46:04.616415 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:04.616254    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2b6d9"
Sep 16 18:46:08.878795 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.878759    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"]
Sep 16 18:46:08.879274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.879111    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerName="util"
Sep 16 18:46:08.879274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.879128    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerName="util"
Sep 16 18:46:08.879274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.879138    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerName="pull"
Sep 16 18:46:08.879274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.879146    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerName="pull"
Sep 16 18:46:08.879274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.879154    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerName="extract"
Sep 16 18:46:08.879274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.879162    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerName="extract"
Sep 16 18:46:08.879274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.879230    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c0d68f-7af6-45a7-b733-5ea438a086e1" containerName="extract"
Sep 16 18:46:08.881024 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.880989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:08.887916 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.887898    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:46:08.888041 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.887938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-b4lhc\""
Sep 16 18:46:08.888239 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.888224    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:46:08.888443 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.888430    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:46:08.904282 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.904255    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"]
Sep 16 18:46:08.911247 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.911224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxts9\" (UniqueName: \"kubernetes.io/projected/0f97f38c-33b4-4003-a787-4c5c03708a7f-kube-api-access-hxts9\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t9gv7\" (UID: \"0f97f38c-33b4-4003-a787-4c5c03708a7f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:08.911342 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:08.911261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0f97f38c-33b4-4003-a787-4c5c03708a7f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t9gv7\" (UID: \"0f97f38c-33b4-4003-a787-4c5c03708a7f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:09.011547 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:09.011511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxts9\" (UniqueName: \"kubernetes.io/projected/0f97f38c-33b4-4003-a787-4c5c03708a7f-kube-api-access-hxts9\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t9gv7\" (UID: \"0f97f38c-33b4-4003-a787-4c5c03708a7f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:09.011707 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:09.011554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0f97f38c-33b4-4003-a787-4c5c03708a7f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t9gv7\" (UID: \"0f97f38c-33b4-4003-a787-4c5c03708a7f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:09.014110 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:09.014086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0f97f38c-33b4-4003-a787-4c5c03708a7f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t9gv7\" (UID: \"0f97f38c-33b4-4003-a787-4c5c03708a7f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:09.022210 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:09.022186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxts9\" (UniqueName: \"kubernetes.io/projected/0f97f38c-33b4-4003-a787-4c5c03708a7f-kube-api-access-hxts9\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-t9gv7\" (UID: \"0f97f38c-33b4-4003-a787-4c5c03708a7f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:09.190518 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:09.190435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:09.470731 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:09.470694    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"]
Sep 16 18:46:09.472893 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:46:09.472863    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f97f38c_33b4_4003_a787_4c5c03708a7f.slice/crio-d29a7c02111124d3166ad9d8cff3b8e910195b040a8c41d5ffd16c9e821cb2bd WatchSource:0}: Error finding container d29a7c02111124d3166ad9d8cff3b8e910195b040a8c41d5ffd16c9e821cb2bd: Status 404 returned error can't find the container with id d29a7c02111124d3166ad9d8cff3b8e910195b040a8c41d5ffd16c9e821cb2bd
Sep 16 18:46:09.634147 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:09.634113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7" event={"ID":"0f97f38c-33b4-4003-a787-4c5c03708a7f","Type":"ContainerStarted","Data":"d29a7c02111124d3166ad9d8cff3b8e910195b040a8c41d5ffd16c9e821cb2bd"}
Sep 16 18:46:12.630295 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.630259    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-zkzld"]
Sep 16 18:46:12.632385 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.632366    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:12.635940 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.635914    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:46:12.636070 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.635977    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-4zrhd\""
Sep 16 18:46:12.636070 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.635928    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:46:12.645160 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.645134    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-zkzld"]
Sep 16 18:46:12.648642 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.648610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7" event={"ID":"0f97f38c-33b4-4003-a787-4c5c03708a7f","Type":"ContainerStarted","Data":"af8a0d04200d651e6144aca36fb1236806d9258c7d4f3c2f82a44a23318214d0"}
Sep 16 18:46:12.648781 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.648768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:12.684679 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.684634    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7" podStartSLOduration=2.246721043 podStartE2EDuration="4.684607448s" totalImagesPullingTime="2.437886405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:12.683175137 +0000 UTC m=+325.813721849" watchObservedRunningTime="2026-09-16 18:46:12.684607448 +0000 UTC m=+325.815154211"
Sep 16 18:46:12.739911 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.739878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgwd\" (UniqueName: \"kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-kube-api-access-7mgwd\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:12.740061 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.739927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:12.740061 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.740002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-cabundle0\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:12.841308 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.841278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgwd\" (UniqueName: \"kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-kube-api-access-7mgwd\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:12.841485 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.841319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:12.841485 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.841358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-cabundle0\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:12.841485 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:46:12.841467    2578 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 18:46:12.841647 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:46:12.841492    2578 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:46:12.841647 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:46:12.841505    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:46:12.841647 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:46:12.841521    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-zkzld: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:46:12.841647 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:46:12.841581    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates podName:4bdf57c9-a032-4d0f-b897-b81fffa2a7c2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.341559377 +0000 UTC m=+326.472106079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates") pod "keda-operator-579c8958f7-zkzld" (UID: "4bdf57c9-a032-4d0f-b897-b81fffa2a7c2") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:46:12.842120 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.842101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-cabundle0\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:12.852677 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:12.852651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgwd\" (UniqueName: \"kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-kube-api-access-7mgwd\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:13.019785 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.019751    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"]
Sep 16 18:46:13.021765 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.021748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.025405 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.025381    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:46:13.030540 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.030513    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"]
Sep 16 18:46:13.143435 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.143402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1b0bcb35-ded7-497f-bea9-d2cc40525dba-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.143609 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.143453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdf5\" (UniqueName: \"kubernetes.io/projected/1b0bcb35-ded7-497f-bea9-d2cc40525dba-kube-api-access-fpdf5\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.143609 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.143553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b0bcb35-ded7-497f-bea9-d2cc40525dba-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.244634 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.244593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1b0bcb35-ded7-497f-bea9-d2cc40525dba-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.244634 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.244646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdf5\" (UniqueName: \"kubernetes.io/projected/1b0bcb35-ded7-497f-bea9-d2cc40525dba-kube-api-access-fpdf5\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.244881 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.244730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b0bcb35-ded7-497f-bea9-d2cc40525dba-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.244982 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.244960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1b0bcb35-ded7-497f-bea9-d2cc40525dba-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.247507 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.247484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b0bcb35-ded7-497f-bea9-d2cc40525dba-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.255116 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.255096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdf5\" (UniqueName: \"kubernetes.io/projected/1b0bcb35-ded7-497f-bea9-d2cc40525dba-kube-api-access-fpdf5\") pod \"keda-metrics-apiserver-77857b7b98-dsb5h\" (UID: \"1b0bcb35-ded7-497f-bea9-d2cc40525dba\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.331905 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.331825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:13.345716 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.345687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:13.345859 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:46:13.345840    2578 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 18:46:13.345900 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:46:13.345872    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-zkzld: secret "keda-operator-certs" not found
Sep 16 18:46:13.345934 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:46:13.345924    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates podName:4bdf57c9-a032-4d0f-b897-b81fffa2a7c2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.345910786 +0000 UTC m=+327.476457479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates") pod "keda-operator-579c8958f7-zkzld" (UID: "4bdf57c9-a032-4d0f-b897-b81fffa2a7c2") : secret "keda-operator-certs" not found
Sep 16 18:46:13.600588 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.600563    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"]
Sep 16 18:46:13.603247 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:46:13.603218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0bcb35_ded7_497f_bea9_d2cc40525dba.slice/crio-260b5fd4ee97d42d97bac7b3d246841c56b1e3b6e61d23ac9bb0c0f878ba87f1 WatchSource:0}: Error finding container 260b5fd4ee97d42d97bac7b3d246841c56b1e3b6e61d23ac9bb0c0f878ba87f1: Status 404 returned error can't find the container with id 260b5fd4ee97d42d97bac7b3d246841c56b1e3b6e61d23ac9bb0c0f878ba87f1
Sep 16 18:46:13.653506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:13.653477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h" event={"ID":"1b0bcb35-ded7-497f-bea9-d2cc40525dba","Type":"ContainerStarted","Data":"260b5fd4ee97d42d97bac7b3d246841c56b1e3b6e61d23ac9bb0c0f878ba87f1"}
Sep 16 18:46:14.353542 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:14.353499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:14.356047 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:14.356028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bdf57c9-a032-4d0f-b897-b81fffa2a7c2-certificates\") pod \"keda-operator-579c8958f7-zkzld\" (UID: \"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2\") " pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:14.442150 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:14.442119    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:14.721456 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:14.721381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-zkzld"]
Sep 16 18:46:14.726235 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:46:14.726195    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdf57c9_a032_4d0f_b897_b81fffa2a7c2.slice/crio-74958969437c9d4eacf1e04005ba8dc337d02989b38eea7e95b8695e8ed9e846 WatchSource:0}: Error finding container 74958969437c9d4eacf1e04005ba8dc337d02989b38eea7e95b8695e8ed9e846: Status 404 returned error can't find the container with id 74958969437c9d4eacf1e04005ba8dc337d02989b38eea7e95b8695e8ed9e846
Sep 16 18:46:15.663918 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:15.663883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-zkzld" event={"ID":"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2","Type":"ContainerStarted","Data":"74958969437c9d4eacf1e04005ba8dc337d02989b38eea7e95b8695e8ed9e846"}
Sep 16 18:46:16.670987 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:16.670897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h" event={"ID":"1b0bcb35-ded7-497f-bea9-d2cc40525dba","Type":"ContainerStarted","Data":"dba912c487007e3961c2377d5d9023f0bdcaf1e6eff65152e13cb4ef87a3de14"}
Sep 16 18:46:16.671426 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:16.671024    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:16.693901 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:16.693851    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h" podStartSLOduration=1.950862814 podStartE2EDuration="4.693835224s" totalImagesPullingTime="2.74297241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:16.692423962 +0000 UTC m=+329.822970665" watchObservedRunningTime="2026-09-16 18:46:16.693835224 +0000 UTC m=+329.824381928"
Sep 16 18:46:18.680890 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:18.680855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-zkzld" event={"ID":"4bdf57c9-a032-4d0f-b897-b81fffa2a7c2","Type":"ContainerStarted","Data":"e362c4814d68bec959c5db1d761bd417a5bd3d96ec71b3adf5738d1fb8ae241b"}
Sep 16 18:46:18.681328 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:18.680988    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:46:18.702714 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:18.702670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-zkzld" podStartSLOduration=3.554869949 podStartE2EDuration="6.702656803s" totalImagesPullingTime="3.147786854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:18.701704442 +0000 UTC m=+331.832251153" watchObservedRunningTime="2026-09-16 18:46:18.702656803 +0000 UTC m=+331.833203514"
Sep 16 18:46:27.681136 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:27.680916    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dsb5h"
Sep 16 18:46:33.656297 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:33.656266    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-t9gv7"
Sep 16 18:46:39.687404 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:46:39.687374    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-zkzld"
Sep 16 18:47:35.427066 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.427031    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-jxpjh"]
Sep 16 18:47:35.429217 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.429200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:35.432181 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.432157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4rtmh\""
Sep 16 18:47:35.432346 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.432242    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:47:35.432346 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.432335    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:47:35.432994 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.432975    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:47:35.439632 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.439607    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jxpjh"]
Sep 16 18:47:35.545334 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.545301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/14dde8ef-ab27-47bd-b0af-c619bfabb0a3-data\") pod \"seaweedfs-64568bcd49-jxpjh\" (UID: \"14dde8ef-ab27-47bd-b0af-c619bfabb0a3\") " pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:35.545520 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.545346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bqn\" (UniqueName: \"kubernetes.io/projected/14dde8ef-ab27-47bd-b0af-c619bfabb0a3-kube-api-access-j2bqn\") pod \"seaweedfs-64568bcd49-jxpjh\" (UID: \"14dde8ef-ab27-47bd-b0af-c619bfabb0a3\") " pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:35.646114 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.646073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/14dde8ef-ab27-47bd-b0af-c619bfabb0a3-data\") pod \"seaweedfs-64568bcd49-jxpjh\" (UID: \"14dde8ef-ab27-47bd-b0af-c619bfabb0a3\") " pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:35.646269 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.646121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bqn\" (UniqueName: \"kubernetes.io/projected/14dde8ef-ab27-47bd-b0af-c619bfabb0a3-kube-api-access-j2bqn\") pod \"seaweedfs-64568bcd49-jxpjh\" (UID: \"14dde8ef-ab27-47bd-b0af-c619bfabb0a3\") " pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:35.646427 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.646407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/14dde8ef-ab27-47bd-b0af-c619bfabb0a3-data\") pod \"seaweedfs-64568bcd49-jxpjh\" (UID: \"14dde8ef-ab27-47bd-b0af-c619bfabb0a3\") " pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:35.655528 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.655503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bqn\" (UniqueName: \"kubernetes.io/projected/14dde8ef-ab27-47bd-b0af-c619bfabb0a3-kube-api-access-j2bqn\") pod \"seaweedfs-64568bcd49-jxpjh\" (UID: \"14dde8ef-ab27-47bd-b0af-c619bfabb0a3\") " pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:35.738529 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:35.738498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:36.012570 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:36.012485    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jxpjh"]
Sep 16 18:47:36.016313 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:47:36.016279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dde8ef_ab27_47bd_b0af_c619bfabb0a3.slice/crio-614812fd03408bbcf578645ba3e67744a84a821432d499763351276d42071860 WatchSource:0}: Error finding container 614812fd03408bbcf578645ba3e67744a84a821432d499763351276d42071860: Status 404 returned error can't find the container with id 614812fd03408bbcf578645ba3e67744a84a821432d499763351276d42071860
Sep 16 18:47:36.966141 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:36.966094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jxpjh" event={"ID":"14dde8ef-ab27-47bd-b0af-c619bfabb0a3","Type":"ContainerStarted","Data":"614812fd03408bbcf578645ba3e67744a84a821432d499763351276d42071860"}
Sep 16 18:47:38.976372 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:38.976335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jxpjh" event={"ID":"14dde8ef-ab27-47bd-b0af-c619bfabb0a3","Type":"ContainerStarted","Data":"af8a604f805e3b9b04d3c5837843df0ae1e86d8122dc1b2530a05e954ed46cb1"}
Sep 16 18:47:38.976795 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:38.976422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:47:38.995455 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:38.995403    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-jxpjh" podStartSLOduration=1.1285757 podStartE2EDuration="3.995384329s" totalImagesPullingTime="2.866808629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:38.994573198 +0000 UTC m=+412.125119923" watchObservedRunningTime="2026-09-16 18:47:38.995384329 +0000 UTC m=+412.125931039"
Sep 16 18:47:44.987540 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:47:44.987510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-jxpjh"
Sep 16 18:48:53.407160 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.407118    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-n8xsh"]
Sep 16 18:48:53.409384 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.409364    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-88c82"]
Sep 16 18:48:53.409552 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.409502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:53.411497 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.411477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:53.413782 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.413764    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:48:53.414151 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.414136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-k7gjj\""
Sep 16 18:48:53.417174 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.417148    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:48:53.417174 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.417168    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:48:53.417789 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.417775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jk96n\""
Sep 16 18:48:53.435939 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.435912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-88c82"]
Sep 16 18:48:53.450310 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.450271    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-n8xsh"]
Sep 16 18:48:53.516716 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.516687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflr4\" (UniqueName: \"kubernetes.io/projected/952661c3-9ea2-4382-a8d0-f09a5147e19c-kube-api-access-wflr4\") pod \"model-serving-api-7b954ffc95-n8xsh\" (UID: \"952661c3-9ea2-4382-a8d0-f09a5147e19c\") " pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:53.516716 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.516721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-cert\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:53.516926 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.516742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-metrics-certs\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:53.516926 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.516842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj669\" (UniqueName: \"kubernetes.io/projected/a6f6d643-53b3-4782-a1f4-087b0a972fa0-kube-api-access-cj669\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:53.516926 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.516882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952661c3-9ea2-4382-a8d0-f09a5147e19c-tls-certs\") pod \"model-serving-api-7b954ffc95-n8xsh\" (UID: \"952661c3-9ea2-4382-a8d0-f09a5147e19c\") " pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:53.618264 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.618233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952661c3-9ea2-4382-a8d0-f09a5147e19c-tls-certs\") pod \"model-serving-api-7b954ffc95-n8xsh\" (UID: \"952661c3-9ea2-4382-a8d0-f09a5147e19c\") " pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:53.618458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.618284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflr4\" (UniqueName: \"kubernetes.io/projected/952661c3-9ea2-4382-a8d0-f09a5147e19c-kube-api-access-wflr4\") pod \"model-serving-api-7b954ffc95-n8xsh\" (UID: \"952661c3-9ea2-4382-a8d0-f09a5147e19c\") " pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:53.618458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.618313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-cert\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:53.618458 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.618336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-metrics-certs\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:53.618458 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:48:53.618394    2578 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 18:48:53.618689 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:48:53.618475    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:48:53.618689 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:48:53.618495    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/952661c3-9ea2-4382-a8d0-f09a5147e19c-tls-certs podName:952661c3-9ea2-4382-a8d0-f09a5147e19c nodeName:}" failed. No retries permitted until 2026-09-16 18:48:54.11845737 +0000 UTC m=+487.249004074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/952661c3-9ea2-4382-a8d0-f09a5147e19c-tls-certs") pod "model-serving-api-7b954ffc95-n8xsh" (UID: "952661c3-9ea2-4382-a8d0-f09a5147e19c") : secret "model-serving-api-tls" not found
Sep 16 18:48:53.618689 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.618400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj669\" (UniqueName: \"kubernetes.io/projected/a6f6d643-53b3-4782-a1f4-087b0a972fa0-kube-api-access-cj669\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:53.618689 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:48:53.618547    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-cert podName:a6f6d643-53b3-4782-a1f4-087b0a972fa0 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:54.118527596 +0000 UTC m=+487.249074291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-cert") pod "odh-model-controller-7b966d7984-88c82" (UID: "a6f6d643-53b3-4782-a1f4-087b0a972fa0") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:48:53.618689 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:48:53.618599    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 16 18:48:53.618689 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:48:53.618650    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-metrics-certs podName:a6f6d643-53b3-4782-a1f4-087b0a972fa0 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:54.118635053 +0000 UTC m=+487.249181771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-metrics-certs") pod "odh-model-controller-7b966d7984-88c82" (UID: "a6f6d643-53b3-4782-a1f4-087b0a972fa0") : secret "odh-model-controller-metrics-tls" not found
Sep 16 18:48:53.630084 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.630059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflr4\" (UniqueName: \"kubernetes.io/projected/952661c3-9ea2-4382-a8d0-f09a5147e19c-kube-api-access-wflr4\") pod \"model-serving-api-7b954ffc95-n8xsh\" (UID: \"952661c3-9ea2-4382-a8d0-f09a5147e19c\") " pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:53.630205 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:53.630102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj669\" (UniqueName: \"kubernetes.io/projected/a6f6d643-53b3-4782-a1f4-087b0a972fa0-kube-api-access-cj669\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:54.123505 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.123470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952661c3-9ea2-4382-a8d0-f09a5147e19c-tls-certs\") pod \"model-serving-api-7b954ffc95-n8xsh\" (UID: \"952661c3-9ea2-4382-a8d0-f09a5147e19c\") " pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:54.123736 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.123522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-cert\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:54.123736 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.123552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-metrics-certs\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:54.126227 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.126204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-metrics-certs\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:54.126643 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.126625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6f6d643-53b3-4782-a1f4-087b0a972fa0-cert\") pod \"odh-model-controller-7b966d7984-88c82\" (UID: \"a6f6d643-53b3-4782-a1f4-087b0a972fa0\") " pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:54.126772 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.126751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952661c3-9ea2-4382-a8d0-f09a5147e19c-tls-certs\") pod \"model-serving-api-7b954ffc95-n8xsh\" (UID: \"952661c3-9ea2-4382-a8d0-f09a5147e19c\") " pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:54.320527 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.320489    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:54.326678 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.326653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:54.600775 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.600742    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-88c82"]
Sep 16 18:48:54.603931 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:54.603906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-n8xsh"]
Sep 16 18:48:54.604170 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:48:54.604143    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f6d643_53b3_4782_a1f4_087b0a972fa0.slice/crio-b256d926480c338f8ccfa72beaa526a4b226f9ca09c3333840f7dd05ac42ca47 WatchSource:0}: Error finding container b256d926480c338f8ccfa72beaa526a4b226f9ca09c3333840f7dd05ac42ca47: Status 404 returned error can't find the container with id b256d926480c338f8ccfa72beaa526a4b226f9ca09c3333840f7dd05ac42ca47
Sep 16 18:48:54.605851 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:48:54.605828    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952661c3_9ea2_4382_a8d0_f09a5147e19c.slice/crio-33efc50aa8938af60b237a9b05fc39f1be704c7522ff75d18a7be533e86fb7e4 WatchSource:0}: Error finding container 33efc50aa8938af60b237a9b05fc39f1be704c7522ff75d18a7be533e86fb7e4: Status 404 returned error can't find the container with id 33efc50aa8938af60b237a9b05fc39f1be704c7522ff75d18a7be533e86fb7e4
Sep 16 18:48:55.254326 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:55.254289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-n8xsh" event={"ID":"952661c3-9ea2-4382-a8d0-f09a5147e19c","Type":"ContainerStarted","Data":"33efc50aa8938af60b237a9b05fc39f1be704c7522ff75d18a7be533e86fb7e4"}
Sep 16 18:48:55.257355 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:55.257308    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-88c82" event={"ID":"a6f6d643-53b3-4782-a1f4-087b0a972fa0","Type":"ContainerStarted","Data":"b256d926480c338f8ccfa72beaa526a4b226f9ca09c3333840f7dd05ac42ca47"}
Sep 16 18:48:58.272274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:58.272235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-n8xsh" event={"ID":"952661c3-9ea2-4382-a8d0-f09a5147e19c","Type":"ContainerStarted","Data":"589386517d81b7efbdc14f5b2a142ca1117cde4cc7817a3d2608d26b7e39b745"}
Sep 16 18:48:58.272745 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:58.272412    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:48:58.274127 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:58.274102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-88c82" event={"ID":"a6f6d643-53b3-4782-a1f4-087b0a972fa0","Type":"ContainerStarted","Data":"f0c32b2d31be88343914508c0a99ba3630bd24126167a37fde912202878cf574"}
Sep 16 18:48:58.274333 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:58.274234    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:48:58.329302 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:48:58.329254    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-n8xsh" podStartSLOduration=1.79996364 podStartE2EDuration="5.329236915s" totalImagesPullingTime="3.529273275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:58.295715045 +0000 UTC m=+491.426261757" watchObservedRunningTime="2026-09-16 18:48:58.329236915 +0000 UTC m=+491.459783628"
Sep 16 18:49:09.280700 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:49:09.280661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-88c82"
Sep 16 18:49:09.282350 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:49:09.282325    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-n8xsh"
Sep 16 18:49:09.301131 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:49:09.301084    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-88c82" podStartSLOduration=12.761184591 podStartE2EDuration="16.301070341s" totalImagesPullingTime="3.53988575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:58.329113133 +0000 UTC m=+491.459659844" watchObservedRunningTime="2026-09-16 18:49:09.301070341 +0000 UTC m=+502.431617052"
Sep 16 18:50:40.254042 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.253988    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"]
Sep 16 18:50:40.256709 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.256690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.259496 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.259466    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:50:40.259622 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.259602    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:50:40.259688 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.259602    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:50:40.259896 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.259877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkvz5\""
Sep 16 18:50:40.259974 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.259904    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-6a476-predictor-serving-cert\""
Sep 16 18:50:40.259974 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.259918    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:50:40.259974 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.259926    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-6a476-kube-rbac-proxy-sar-config\""
Sep 16 18:50:40.268900 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.268879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"]
Sep 16 18:50:40.379482 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.379455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4vj\" (UniqueName: \"kubernetes.io/projected/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-api-access-lt4vj\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.379649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.379505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.379649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.379537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6401a490-6320-4ab6-b412-c86da97ca4d5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.379649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.379583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.379649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.379610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6401a490-6320-4ab6-b412-c86da97ca4d5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.480974 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.480937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.480974 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.480978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6401a490-6320-4ab6-b412-c86da97ca4d5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.481318 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.481025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4vj\" (UniqueName: \"kubernetes.io/projected/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-api-access-lt4vj\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.481318 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.481067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.481318 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.481088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6401a490-6320-4ab6-b412-c86da97ca4d5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.481500 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.481459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6401a490-6320-4ab6-b412-c86da97ca4d5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.481718 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.481688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.481837 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.481816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.483847 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.483829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6401a490-6320-4ab6-b412-c86da97ca4d5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.497579 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.497541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4vj\" (UniqueName: \"kubernetes.io/projected/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-api-access-lt4vj\") pod \"isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.512064 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.511989    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"]
Sep 16 18:50:40.514376 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.514361    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.517001 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.516979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:50:40.517126 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.517026    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:50:40.529254 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.529235    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"]
Sep 16 18:50:40.566561 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.566532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:50:40.682536 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.682507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7caa75b7-a97d-4420-8383-9030897f703b-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.682702 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.682570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caa75b7-a97d-4420-8383-9030897f703b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.682702 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.682593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776x6\" (UniqueName: \"kubernetes.io/projected/7caa75b7-a97d-4420-8383-9030897f703b-kube-api-access-776x6\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.682702 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.682616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.682702 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.682636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.783801 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.783727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caa75b7-a97d-4420-8383-9030897f703b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.783801 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.783759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776x6\" (UniqueName: \"kubernetes.io/projected/7caa75b7-a97d-4420-8383-9030897f703b-kube-api-access-776x6\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.783801 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.783778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.783801 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.783796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.784153 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.783827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7caa75b7-a97d-4420-8383-9030897f703b-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.784236 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.784216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caa75b7-a97d-4420-8383-9030897f703b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.784570 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.784548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.784570 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.784563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.786649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.786629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7caa75b7-a97d-4420-8383-9030897f703b-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.793234 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.793208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-776x6\" (UniqueName: \"kubernetes.io/projected/7caa75b7-a97d-4420-8383-9030897f703b-kube-api-access-776x6\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.826061 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.826035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:50:40.844274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:40.842275    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"]
Sep 16 18:50:41.106790 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:41.106715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"]
Sep 16 18:50:41.109756 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:50:41.109725    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caa75b7_a97d_4420_8383_9030897f703b.slice/crio-2e4808c2c7da6fa77d7c5a3dc82c0154d708ae6e68549e2955690140b1593926 WatchSource:0}: Error finding container 2e4808c2c7da6fa77d7c5a3dc82c0154d708ae6e68549e2955690140b1593926: Status 404 returned error can't find the container with id 2e4808c2c7da6fa77d7c5a3dc82c0154d708ae6e68549e2955690140b1593926
Sep 16 18:50:41.659184 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:41.659093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" event={"ID":"7caa75b7-a97d-4420-8383-9030897f703b","Type":"ContainerStarted","Data":"2e4808c2c7da6fa77d7c5a3dc82c0154d708ae6e68549e2955690140b1593926"}
Sep 16 18:50:41.662263 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:41.662173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerStarted","Data":"9d92a17e397123614899c0ed5979be6df063edee2817f13a217cfc7e3fe4c406"}
Sep 16 18:50:44.681041 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:44.680988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" event={"ID":"7caa75b7-a97d-4420-8383-9030897f703b","Type":"ContainerStarted","Data":"2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e"}
Sep 16 18:50:44.682866 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:44.682835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerStarted","Data":"3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7"}
Sep 16 18:50:47.238278 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:47.238245    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:50:47.239626 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:47.239605    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:50:48.701244 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:48.701152    2578 generic.go:356] "Generic (PLEG): container finished" podID="7caa75b7-a97d-4420-8383-9030897f703b" containerID="2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e" exitCode=0
Sep 16 18:50:48.701244 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:48.701225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" event={"ID":"7caa75b7-a97d-4420-8383-9030897f703b","Type":"ContainerDied","Data":"2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e"}
Sep 16 18:50:48.702957 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:48.702881    2578 generic.go:356] "Generic (PLEG): container finished" podID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerID="3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7" exitCode=0
Sep 16 18:50:48.703053 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:50:48.702961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerDied","Data":"3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7"}
Sep 16 18:51:05.274379 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:05.274355    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:51:05.827649 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:05.827608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" event={"ID":"7caa75b7-a97d-4420-8383-9030897f703b","Type":"ContainerStarted","Data":"cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1"}
Sep 16 18:51:05.833952 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:05.833916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerStarted","Data":"6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819"}
Sep 16 18:51:08.850818 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:08.850721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" event={"ID":"7caa75b7-a97d-4420-8383-9030897f703b","Type":"ContainerStarted","Data":"70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa"}
Sep 16 18:51:08.850818 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:08.850795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:51:08.851323 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:08.850946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:51:08.852719 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:08.852688    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:51:08.853335 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:08.853314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerStarted","Data":"3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434"}
Sep 16 18:51:09.859087 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:09.859044    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:51:10.870831 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:10.870793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerStarted","Data":"7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3"}
Sep 16 18:51:10.871264 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:10.870986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:51:10.871264 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:10.871070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:51:10.871264 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:10.871109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:51:10.872857 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:10.872818    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:51:10.873622 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:10.873597    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:10.892367 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:10.892315    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podStartSLOduration=1.943425024 podStartE2EDuration="30.892300556s" totalImagesPullingTime="25.068375827s" totalInitContainerRuntime="3.880499705s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:40 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:10.891094198 +0000 UTC m=+624.021640900" watchObservedRunningTime="2026-09-16 18:51:10.892300556 +0000 UTC m=+624.022847287"
Sep 16 18:51:10.893463 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:10.893425    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podStartSLOduration=4.582188828 podStartE2EDuration="30.893410669s" totalImagesPullingTime="22.55373216s" totalInitContainerRuntime="3.757489681s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:08.871682699 +0000 UTC m=+622.002229410" watchObservedRunningTime="2026-09-16 18:51:10.893410669 +0000 UTC m=+624.023957392"
Sep 16 18:51:11.875391 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:11.875350    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:51:11.875790 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:11.875764    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:14.864411 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:14.864379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:51:14.865143 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:14.865117    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:51:16.879402 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:16.879374    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:51:16.879897 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:16.879870    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:51:16.880354 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:16.880333    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:24.865332 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:24.865291    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:51:26.880494 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:26.880449    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:51:26.880944 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:26.880918    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:34.864989 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:34.864951    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:51:36.879981 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:36.879937    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:51:36.880494 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:36.880233    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:44.865621 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:44.865581    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:51:46.879793 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:46.879752    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:51:46.880262 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:46.880239    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:54.865451 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:54.865405    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:51:56.880397 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:56.880352    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:51:56.880856 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:51:56.880786    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:04.865610 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:04.865570    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:52:06.880032 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:06.879947    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:52:06.880519 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:06.880481    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:14.865274 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:14.865230    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 16 18:52:16.879909 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:16.879847    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:52:16.880326 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:16.880185    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:20.623816 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:20.623774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"]
Sep 16 18:52:20.624248 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:20.624063    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container" containerID="cri-o://cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1" gracePeriod=30
Sep 16 18:52:20.624248 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:20.624074    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kube-rbac-proxy" containerID="cri-o://70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa" gracePeriod=30
Sep 16 18:52:20.625224 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:20.625201    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:52:21.167876 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:21.167841    2578 generic.go:356] "Generic (PLEG): container finished" podID="7caa75b7-a97d-4420-8383-9030897f703b" containerID="70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa" exitCode=2
Sep 16 18:52:21.168074 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:21.167879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" event={"ID":"7caa75b7-a97d-4420-8383-9030897f703b","Type":"ContainerDied","Data":"70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa"}
Sep 16 18:52:21.331897 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:21.331869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:52:21.332081 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:21.331968    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:52:24.859872 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:24.859829    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 16 18:52:25.564393 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.564370    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:52:25.617644 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.617614    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-kube-rbac-proxy-sar-config\") pod \"7caa75b7-a97d-4420-8383-9030897f703b\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") "
Sep 16 18:52:25.617834 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.617680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caa75b7-a97d-4420-8383-9030897f703b-kserve-provision-location\") pod \"7caa75b7-a97d-4420-8383-9030897f703b\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") "
Sep 16 18:52:25.617834 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.617699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-cabundle-cert\") pod \"7caa75b7-a97d-4420-8383-9030897f703b\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") "
Sep 16 18:52:25.617834 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.617726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7caa75b7-a97d-4420-8383-9030897f703b-proxy-tls\") pod \"7caa75b7-a97d-4420-8383-9030897f703b\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") "
Sep 16 18:52:25.617834 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.617754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-776x6\" (UniqueName: \"kubernetes.io/projected/7caa75b7-a97d-4420-8383-9030897f703b-kube-api-access-776x6\") pod \"7caa75b7-a97d-4420-8383-9030897f703b\" (UID: \"7caa75b7-a97d-4420-8383-9030897f703b\") "
Sep 16 18:52:25.618081 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.618002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caa75b7-a97d-4420-8383-9030897f703b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7caa75b7-a97d-4420-8383-9030897f703b" (UID: "7caa75b7-a97d-4420-8383-9030897f703b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:25.618081 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.618048    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7caa75b7-a97d-4420-8383-9030897f703b" (UID: "7caa75b7-a97d-4420-8383-9030897f703b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:25.618081 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.618064    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7caa75b7-a97d-4420-8383-9030897f703b" (UID: "7caa75b7-a97d-4420-8383-9030897f703b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:25.620206 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.620181    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7caa75b7-a97d-4420-8383-9030897f703b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7caa75b7-a97d-4420-8383-9030897f703b" (UID: "7caa75b7-a97d-4420-8383-9030897f703b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:25.620206 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.620189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caa75b7-a97d-4420-8383-9030897f703b-kube-api-access-776x6" (OuterVolumeSpecName: "kube-api-access-776x6") pod "7caa75b7-a97d-4420-8383-9030897f703b" (UID: "7caa75b7-a97d-4420-8383-9030897f703b"). InnerVolumeSpecName "kube-api-access-776x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:25.718755 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.718659    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.718755 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.718695    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7caa75b7-a97d-4420-8383-9030897f703b-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.718755 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.718709    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7caa75b7-a97d-4420-8383-9030897f703b-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.718755 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.718722    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7caa75b7-a97d-4420-8383-9030897f703b-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.718755 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:25.718732    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776x6\" (UniqueName: \"kubernetes.io/projected/7caa75b7-a97d-4420-8383-9030897f703b-kube-api-access-776x6\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:52:26.189925 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.189885    2578 generic.go:356] "Generic (PLEG): container finished" podID="7caa75b7-a97d-4420-8383-9030897f703b" containerID="cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1" exitCode=0
Sep 16 18:52:26.190340 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.189947    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"
Sep 16 18:52:26.190340 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.189971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" event={"ID":"7caa75b7-a97d-4420-8383-9030897f703b","Type":"ContainerDied","Data":"cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1"}
Sep 16 18:52:26.190340 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.190059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk" event={"ID":"7caa75b7-a97d-4420-8383-9030897f703b","Type":"ContainerDied","Data":"2e4808c2c7da6fa77d7c5a3dc82c0154d708ae6e68549e2955690140b1593926"}
Sep 16 18:52:26.190340 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.190087    2578 scope.go:117] "RemoveContainer" containerID="70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa"
Sep 16 18:52:26.198718 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.198703    2578 scope.go:117] "RemoveContainer" containerID="cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1"
Sep 16 18:52:26.206550 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.206532    2578 scope.go:117] "RemoveContainer" containerID="2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e"
Sep 16 18:52:26.213864 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.213832    2578 scope.go:117] "RemoveContainer" containerID="70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa"
Sep 16 18:52:26.214221 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:52:26.214198    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa\": container with ID starting with 70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa not found: ID does not exist" containerID="70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa"
Sep 16 18:52:26.214302 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.214236    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa"} err="failed to get container status \"70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa\": rpc error: code = NotFound desc = could not find container \"70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa\": container with ID starting with 70cffc8ec784c255272b676bd958a49eea442dd0e83d8ea65e1c417e73f61efa not found: ID does not exist"
Sep 16 18:52:26.214302 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.214255    2578 scope.go:117] "RemoveContainer" containerID="cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1"
Sep 16 18:52:26.214544 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:52:26.214526    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1\": container with ID starting with cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1 not found: ID does not exist" containerID="cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1"
Sep 16 18:52:26.214593 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.214551    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1"} err="failed to get container status \"cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1\": rpc error: code = NotFound desc = could not find container \"cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1\": container with ID starting with cff2e184ee1e6797b9261ae6b409e2ce0779ea394bbb85a8655185fa1c25e0b1 not found: ID does not exist"
Sep 16 18:52:26.214593 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.214569    2578 scope.go:117] "RemoveContainer" containerID="2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e"
Sep 16 18:52:26.214861 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:52:26.214841    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e\": container with ID starting with 2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e not found: ID does not exist" containerID="2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e"
Sep 16 18:52:26.214920 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.214870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e"} err="failed to get container status \"2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e\": rpc error: code = NotFound desc = could not find container \"2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e\": container with ID starting with 2e07c5e3fe38342bdcb73e26a3875cacbd2cc058077c7d29a7aeeb1d0ddd652e not found: ID does not exist"
Sep 16 18:52:26.215182 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.215166    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"]
Sep 16 18:52:26.220398 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:26.220379    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-tt4mk"]
Sep 16 18:52:27.336401 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:27.336352    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7caa75b7-a97d-4420-8383-9030897f703b" path="/var/lib/kubelet/pods/7caa75b7-a97d-4420-8383-9030897f703b/volumes"
Sep 16 18:52:30.262986 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.262949    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"]
Sep 16 18:52:30.263362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.263251    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kube-rbac-proxy"
Sep 16 18:52:30.263362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.263262    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kube-rbac-proxy"
Sep 16 18:52:30.263362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.263271    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="storage-initializer"
Sep 16 18:52:30.263362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.263276    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="storage-initializer"
Sep 16 18:52:30.263362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.263284    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container"
Sep 16 18:52:30.263362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.263290    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container"
Sep 16 18:52:30.263362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.263337    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kserve-container"
Sep 16 18:52:30.263362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.263344    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7caa75b7-a97d-4420-8383-9030897f703b" containerName="kube-rbac-proxy"
Sep 16 18:52:30.267846 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.267828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.270598 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.270580    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:52:30.271456 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.271436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:52:30.287847 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.287819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"]
Sep 16 18:52:30.351798 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.351766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.351798 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.351809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f6672-4c35-4482-a933-3b78ac5f1c0b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.352103 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.351853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.352103 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.351880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f6672-4c35-4482-a933-3b78ac5f1c0b-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.352103 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.351959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdrl\" (UniqueName: \"kubernetes.io/projected/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-api-access-9cdrl\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.452585 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.452554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.452585 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.452590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f6672-4c35-4482-a933-3b78ac5f1c0b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.452813 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.452617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.452813 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.452741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f6672-4c35-4482-a933-3b78ac5f1c0b-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.452924 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.452816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdrl\" (UniqueName: \"kubernetes.io/projected/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-api-access-9cdrl\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.452994 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.452980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f6672-4c35-4482-a933-3b78ac5f1c0b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.453354 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.453330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.453434 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.453407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.455362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.455344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f6672-4c35-4482-a933-3b78ac5f1c0b-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.464824 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.464802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdrl\" (UniqueName: \"kubernetes.io/projected/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-api-access-9cdrl\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.577184 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.577102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:30.874319 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:30.874232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"]
Sep 16 18:52:30.878096 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:52:30.878061    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988f6672_4c35_4482_a933_3b78ac5f1c0b.slice/crio-85ff4d11ef0f4f80cd562a82e2d2fc0588df442ba5d570efaa0460518ad5b096 WatchSource:0}: Error finding container 85ff4d11ef0f4f80cd562a82e2d2fc0588df442ba5d570efaa0460518ad5b096: Status 404 returned error can't find the container with id 85ff4d11ef0f4f80cd562a82e2d2fc0588df442ba5d570efaa0460518ad5b096
Sep 16 18:52:31.215943 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:31.215837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" event={"ID":"988f6672-4c35-4482-a933-3b78ac5f1c0b","Type":"ContainerStarted","Data":"b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297"}
Sep 16 18:52:31.215943 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:31.215900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" event={"ID":"988f6672-4c35-4482-a933-3b78ac5f1c0b","Type":"ContainerStarted","Data":"85ff4d11ef0f4f80cd562a82e2d2fc0588df442ba5d570efaa0460518ad5b096"}
Sep 16 18:52:35.180489 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:35.180460    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"]
Sep 16 18:52:35.180888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:35.180764    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" containerID="cri-o://6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819" gracePeriod=30
Sep 16 18:52:35.180888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:35.180818    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy" containerID="cri-o://3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434" gracePeriod=30
Sep 16 18:52:35.180888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:35.180773    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" containerID="cri-o://7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3" gracePeriod=30
Sep 16 18:52:35.234221 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:35.234185    2578 generic.go:356] "Generic (PLEG): container finished" podID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerID="b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297" exitCode=0
Sep 16 18:52:35.234364 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:35.234258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" event={"ID":"988f6672-4c35-4482-a933-3b78ac5f1c0b","Type":"ContainerDied","Data":"b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297"}
Sep 16 18:52:36.240840 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.240803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" event={"ID":"988f6672-4c35-4482-a933-3b78ac5f1c0b","Type":"ContainerStarted","Data":"6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16"}
Sep 16 18:52:36.241285 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.240866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" event={"ID":"988f6672-4c35-4482-a933-3b78ac5f1c0b","Type":"ContainerStarted","Data":"26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02"}
Sep 16 18:52:36.241285 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.241184    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:36.241285 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.241238    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:36.242545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.242517    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:52:36.243392 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.243370    2578 generic.go:356] "Generic (PLEG): container finished" podID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerID="3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434" exitCode=2
Sep 16 18:52:36.243502 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.243420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerDied","Data":"3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434"}
Sep 16 18:52:36.262983 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.262937    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podStartSLOduration=2.60304478 podStartE2EDuration="6.262922484s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659877704s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:36.261244736 +0000 UTC m=+709.391791449" watchObservedRunningTime="2026-09-16 18:52:36.262922484 +0000 UTC m=+709.393469194"
Sep 16 18:52:36.876285 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:36.876244    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 16 18:52:37.247782 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:37.247743    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:52:40.262424 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:40.262396    2578 generic.go:356] "Generic (PLEG): container finished" podID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerID="6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819" exitCode=0
Sep 16 18:52:40.262792 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:40.262448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerDied","Data":"6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819"}
Sep 16 18:52:41.327397 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:41.327345    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:52:41.327819 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:41.327688    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:41.875843 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:41.875801    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 16 18:52:42.252734 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:42.252706    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:52:42.253227 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:42.253198    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:52:46.875423 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:46.875378    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 16 18:52:46.875951 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:46.875556    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:52:51.327563 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:51.327516    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:52:51.328089 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:51.327915    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:51.875483 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:51.875439    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 16 18:52:52.253389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:52.253353    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:52:56.875679 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:52:56.875633    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 16 18:53:01.327350 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:01.327301    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 16 18:53:01.327797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:01.327671    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:01.331875 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:01.331851    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:53:01.332036 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:01.331969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:53:01.876328 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:01.876290    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 16 18:53:02.253519 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:02.253485    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:05.329433 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.329409    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:53:05.364742 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.364706    2578 generic.go:356] "Generic (PLEG): container finished" podID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerID="7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3" exitCode=0
Sep 16 18:53:05.364876 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.364780    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"
Sep 16 18:53:05.364876 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.364791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerDied","Data":"7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3"}
Sep 16 18:53:05.364949 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.364888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp" event={"ID":"6401a490-6320-4ab6-b412-c86da97ca4d5","Type":"ContainerDied","Data":"9d92a17e397123614899c0ed5979be6df063edee2817f13a217cfc7e3fe4c406"}
Sep 16 18:53:05.364949 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.364941    2578 scope.go:117] "RemoveContainer" containerID="7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3"
Sep 16 18:53:05.372942 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.372923    2578 scope.go:117] "RemoveContainer" containerID="3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434"
Sep 16 18:53:05.380248 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.380227    2578 scope.go:117] "RemoveContainer" containerID="6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819"
Sep 16 18:53:05.387713 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.387698    2578 scope.go:117] "RemoveContainer" containerID="3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7"
Sep 16 18:53:05.394727 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.394706    2578 scope.go:117] "RemoveContainer" containerID="7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3"
Sep 16 18:53:05.395027 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:53:05.394990    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3\": container with ID starting with 7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3 not found: ID does not exist" containerID="7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3"
Sep 16 18:53:05.395101 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.395035    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3"} err="failed to get container status \"7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3\": rpc error: code = NotFound desc = could not find container \"7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3\": container with ID starting with 7717685f97a684d54f8612e178aacd28f84d5c3cde8dadbb67207b710e8fa6a3 not found: ID does not exist"
Sep 16 18:53:05.395101 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.395061    2578 scope.go:117] "RemoveContainer" containerID="3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434"
Sep 16 18:53:05.395333 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:53:05.395316    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434\": container with ID starting with 3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434 not found: ID does not exist" containerID="3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434"
Sep 16 18:53:05.395389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.395343    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434"} err="failed to get container status \"3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434\": rpc error: code = NotFound desc = could not find container \"3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434\": container with ID starting with 3f9015db1e7f07591ff963a95d5d907d4a28def19a88ea83ad10f79928096434 not found: ID does not exist"
Sep 16 18:53:05.395389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.395359    2578 scope.go:117] "RemoveContainer" containerID="6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819"
Sep 16 18:53:05.395592 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:53:05.395574    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819\": container with ID starting with 6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819 not found: ID does not exist" containerID="6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819"
Sep 16 18:53:05.395631 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.395598    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819"} err="failed to get container status \"6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819\": rpc error: code = NotFound desc = could not find container \"6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819\": container with ID starting with 6f3c48f72d92755ceee781dd0e69460075bb9cf5a9e8ce6cd78153c640825819 not found: ID does not exist"
Sep 16 18:53:05.395631 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.395618    2578 scope.go:117] "RemoveContainer" containerID="3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7"
Sep 16 18:53:05.395816 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:53:05.395796    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7\": container with ID starting with 3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7 not found: ID does not exist" containerID="3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7"
Sep 16 18:53:05.395874 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.395830    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7"} err="failed to get container status \"3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7\": rpc error: code = NotFound desc = could not find container \"3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7\": container with ID starting with 3fefbc51a7b21f81e73f7479b764131623c88abb28ccd95746038cdfd5bbf6b7 not found: ID does not exist"
Sep 16 18:53:05.440880 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.440797    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-cabundle-cert\") pod \"6401a490-6320-4ab6-b412-c86da97ca4d5\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") "
Sep 16 18:53:05.440880 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.440845    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6401a490-6320-4ab6-b412-c86da97ca4d5-kserve-provision-location\") pod \"6401a490-6320-4ab6-b412-c86da97ca4d5\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") "
Sep 16 18:53:05.440880 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.440879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt4vj\" (UniqueName: \"kubernetes.io/projected/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-api-access-lt4vj\") pod \"6401a490-6320-4ab6-b412-c86da97ca4d5\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") "
Sep 16 18:53:05.441155 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.440901    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-rbac-proxy-sar-config\") pod \"6401a490-6320-4ab6-b412-c86da97ca4d5\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") "
Sep 16 18:53:05.441155 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.440936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6401a490-6320-4ab6-b412-c86da97ca4d5-proxy-tls\") pod \"6401a490-6320-4ab6-b412-c86da97ca4d5\" (UID: \"6401a490-6320-4ab6-b412-c86da97ca4d5\") "
Sep 16 18:53:05.441289 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.441259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6401a490-6320-4ab6-b412-c86da97ca4d5" (UID: "6401a490-6320-4ab6-b412-c86da97ca4d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:53:05.441352 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.441296    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6401a490-6320-4ab6-b412-c86da97ca4d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6401a490-6320-4ab6-b412-c86da97ca4d5" (UID: "6401a490-6320-4ab6-b412-c86da97ca4d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:05.441352 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.441321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6401a490-6320-4ab6-b412-c86da97ca4d5" (UID: "6401a490-6320-4ab6-b412-c86da97ca4d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:53:05.443171 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.443145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-api-access-lt4vj" (OuterVolumeSpecName: "kube-api-access-lt4vj") pod "6401a490-6320-4ab6-b412-c86da97ca4d5" (UID: "6401a490-6320-4ab6-b412-c86da97ca4d5"). InnerVolumeSpecName "kube-api-access-lt4vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:05.443271 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.443218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6401a490-6320-4ab6-b412-c86da97ca4d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6401a490-6320-4ab6-b412-c86da97ca4d5" (UID: "6401a490-6320-4ab6-b412-c86da97ca4d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:53:05.542416 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.542377    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6401a490-6320-4ab6-b412-c86da97ca4d5-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:53:05.542416 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.542409    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt4vj\" (UniqueName: \"kubernetes.io/projected/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-api-access-lt4vj\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:53:05.542416 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.542419    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:53:05.542637 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.542428    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6401a490-6320-4ab6-b412-c86da97ca4d5-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:53:05.542637 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.542439    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6401a490-6320-4ab6-b412-c86da97ca4d5-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:53:05.692309 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.692238    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"]
Sep 16 18:53:05.696496 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:05.696474    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6a476-predictor-9c66c784c-gxnwp"]
Sep 16 18:53:07.331924 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:07.331886    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" path="/var/lib/kubelet/pods/6401a490-6320-4ab6-b412-c86da97ca4d5/volumes"
Sep 16 18:53:12.254214 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:12.254176    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:22.253420 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:22.253375    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:32.253900 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:32.253858    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:42.253915 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:42.253876    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:52.254620 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:53:52.254592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:54:00.284634 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:00.284602    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"]
Sep 16 18:54:00.285172 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:00.284873    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" containerID="cri-o://26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02" gracePeriod=30
Sep 16 18:54:00.285172 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:00.284905    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kube-rbac-proxy" containerID="cri-o://6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16" gracePeriod=30
Sep 16 18:54:00.583282 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:00.583204    2578 generic.go:356] "Generic (PLEG): container finished" podID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerID="6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16" exitCode=2
Sep 16 18:54:00.583282 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:00.583272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" event={"ID":"988f6672-4c35-4482-a933-3b78ac5f1c0b","Type":"ContainerDied","Data":"6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16"}
Sep 16 18:54:02.248138 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:02.248092    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 16 18:54:02.253454 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:02.253427    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:54:04.919105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.919079    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:54:04.943923 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.943894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cdrl\" (UniqueName: \"kubernetes.io/projected/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-api-access-9cdrl\") pod \"988f6672-4c35-4482-a933-3b78ac5f1c0b\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") "
Sep 16 18:54:04.944059 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.943978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f6672-4c35-4482-a933-3b78ac5f1c0b-proxy-tls\") pod \"988f6672-4c35-4482-a933-3b78ac5f1c0b\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") "
Sep 16 18:54:04.944144 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.944130    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f6672-4c35-4482-a933-3b78ac5f1c0b-kserve-provision-location\") pod \"988f6672-4c35-4482-a933-3b78ac5f1c0b\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") "
Sep 16 18:54:04.944188 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.944168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-cabundle-cert\") pod \"988f6672-4c35-4482-a933-3b78ac5f1c0b\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") "
Sep 16 18:54:04.944230 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.944216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-rbac-proxy-sar-config\") pod \"988f6672-4c35-4482-a933-3b78ac5f1c0b\" (UID: \"988f6672-4c35-4482-a933-3b78ac5f1c0b\") "
Sep 16 18:54:04.944520 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.944481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988f6672-4c35-4482-a933-3b78ac5f1c0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "988f6672-4c35-4482-a933-3b78ac5f1c0b" (UID: "988f6672-4c35-4482-a933-3b78ac5f1c0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:04.944641 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.944588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "988f6672-4c35-4482-a933-3b78ac5f1c0b" (UID: "988f6672-4c35-4482-a933-3b78ac5f1c0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:04.945001 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.944969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "988f6672-4c35-4482-a933-3b78ac5f1c0b" (UID: "988f6672-4c35-4482-a933-3b78ac5f1c0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:04.946177 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.946148    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-api-access-9cdrl" (OuterVolumeSpecName: "kube-api-access-9cdrl") pod "988f6672-4c35-4482-a933-3b78ac5f1c0b" (UID: "988f6672-4c35-4482-a933-3b78ac5f1c0b"). InnerVolumeSpecName "kube-api-access-9cdrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:04.946405 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:04.946387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988f6672-4c35-4482-a933-3b78ac5f1c0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "988f6672-4c35-4482-a933-3b78ac5f1c0b" (UID: "988f6672-4c35-4482-a933-3b78ac5f1c0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:05.045242 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.045214    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f6672-4c35-4482-a933-3b78ac5f1c0b-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:54:05.045242 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.045241    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:54:05.045426 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.045249    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:54:05.045426 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.045258    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdrl\" (UniqueName: \"kubernetes.io/projected/988f6672-4c35-4482-a933-3b78ac5f1c0b-kube-api-access-9cdrl\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:54:05.045426 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.045270    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f6672-4c35-4482-a933-3b78ac5f1c0b-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:54:05.462172 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462139    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"]
Sep 16 18:54:05.462422 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462409    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy"
Sep 16 18:54:05.462469 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462424    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy"
Sep 16 18:54:05.462469 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462435    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container"
Sep 16 18:54:05.462469 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container"
Sep 16 18:54:05.462469 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462453    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="storage-initializer"
Sep 16 18:54:05.462469 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462459    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="storage-initializer"
Sep 16 18:54:05.462469 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462465    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="storage-initializer"
Sep 16 18:54:05.462469 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462471    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="storage-initializer"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462478    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462484    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462493    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kube-rbac-proxy"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462498    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kube-rbac-proxy"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462503    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462508    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462548    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kserve-container"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462554    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kube-rbac-proxy"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462560    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="agent"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462566    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerName="kserve-container"
Sep 16 18:54:05.462669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.462573    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6401a490-6320-4ab6-b412-c86da97ca4d5" containerName="kube-rbac-proxy"
Sep 16 18:54:05.465715 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.465696    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.468442 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.468423    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 18:54:05.468685 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.468665    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 18:54:05.477587 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.477566    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"]
Sep 16 18:54:05.500689 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.500665    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"]
Sep 16 18:54:05.503909 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.503894    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.506266 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.506249    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 18:54:05.514690 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.514671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"]
Sep 16 18:54:05.549731 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.549702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5mg\" (UniqueName: \"kubernetes.io/projected/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-api-access-tn5mg\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.549884 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.549739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.549884 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.549768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.549884 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.549858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46x6z\" (UniqueName: \"kubernetes.io/projected/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-api-access-46x6z\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.549986 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.549887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.549986 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.549956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32135d58-9b0f-4b93-bdd7-9b0068782b3b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.550081 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.549998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.550081 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.550044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.550142 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.550109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.550142 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.550132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.604431 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.604355    2578 generic.go:356] "Generic (PLEG): container finished" podID="988f6672-4c35-4482-a933-3b78ac5f1c0b" containerID="26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02" exitCode=0
Sep 16 18:54:05.604431 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.604415    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"
Sep 16 18:54:05.604652 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.604437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" event={"ID":"988f6672-4c35-4482-a933-3b78ac5f1c0b","Type":"ContainerDied","Data":"26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02"}
Sep 16 18:54:05.604652 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.604498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz" event={"ID":"988f6672-4c35-4482-a933-3b78ac5f1c0b","Type":"ContainerDied","Data":"85ff4d11ef0f4f80cd562a82e2d2fc0588df442ba5d570efaa0460518ad5b096"}
Sep 16 18:54:05.604652 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.604526    2578 scope.go:117] "RemoveContainer" containerID="6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16"
Sep 16 18:54:05.612556 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.612540    2578 scope.go:117] "RemoveContainer" containerID="26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02"
Sep 16 18:54:05.621580 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.621564    2578 scope.go:117] "RemoveContainer" containerID="b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297"
Sep 16 18:54:05.624591 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.624572    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"]
Sep 16 18:54:05.628988 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.628973    2578 scope.go:117] "RemoveContainer" containerID="6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16"
Sep 16 18:54:05.629312 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:54:05.629287    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16\": container with ID starting with 6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16 not found: ID does not exist" containerID="6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16"
Sep 16 18:54:05.629387 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.629320    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16"} err="failed to get container status \"6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16\": rpc error: code = NotFound desc = could not find container \"6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16\": container with ID starting with 6bf56763a9af71f3af405ed6dd1b4c29088272845ef04e45fa7236784d38da16 not found: ID does not exist"
Sep 16 18:54:05.629387 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.629336    2578 scope.go:117] "RemoveContainer" containerID="26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02"
Sep 16 18:54:05.629591 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:54:05.629573    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02\": container with ID starting with 26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02 not found: ID does not exist" containerID="26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02"
Sep 16 18:54:05.629652 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.629595    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02"} err="failed to get container status \"26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02\": rpc error: code = NotFound desc = could not find container \"26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02\": container with ID starting with 26b156ec7f61311e931616ef1725706eb0a941fd6d92d83a2b55bc32f71b3f02 not found: ID does not exist"
Sep 16 18:54:05.629652 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.629609    2578 scope.go:117] "RemoveContainer" containerID="b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297"
Sep 16 18:54:05.629864 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:54:05.629845    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297\": container with ID starting with b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297 not found: ID does not exist" containerID="b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297"
Sep 16 18:54:05.629864 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.629867    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297"} err="failed to get container status \"b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297\": rpc error: code = NotFound desc = could not find container \"b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297\": container with ID starting with b22768050f343242dbb3d59ea828ef528dae6974ca5042b8ac4274f761715297 not found: ID does not exist"
Sep 16 18:54:05.633768 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.633749    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-qp9dz"]
Sep 16 18:54:05.651468 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32135d58-9b0f-4b93-bdd7-9b0068782b3b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.651588 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.651588 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.651588 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.651588 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.651792 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5mg\" (UniqueName: \"kubernetes.io/projected/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-api-access-tn5mg\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.651792 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.651792 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.651792 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46x6z\" (UniqueName: \"kubernetes.io/projected/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-api-access-46x6z\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.651792 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.652080 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.651946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.652196 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.652176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.652644 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.652296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.652644 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.652439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.652644 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.652588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.652938 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.652917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.654028 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.653988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32135d58-9b0f-4b93-bdd7-9b0068782b3b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.654361 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.654343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.661653 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.661627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46x6z\" (UniqueName: \"kubernetes.io/projected/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-api-access-46x6z\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-knzqz\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:05.661791 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.661773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5mg\" (UniqueName: \"kubernetes.io/projected/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-api-access-tn5mg\") pod \"isvc-canary-stop-v2-predictor-854695c447-h74zz\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.775849 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.775811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:05.814140 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:05.814078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:06.050791 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:06.050768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"]
Sep 16 18:54:06.053275 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:54:06.053247    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32135d58_9b0f_4b93_bdd7_9b0068782b3b.slice/crio-e27645463086ceb295e5655b8a4e695465e4aef5c8b9fb1709bd8d8e117deb9f WatchSource:0}: Error finding container e27645463086ceb295e5655b8a4e695465e4aef5c8b9fb1709bd8d8e117deb9f: Status 404 returned error can't find the container with id e27645463086ceb295e5655b8a4e695465e4aef5c8b9fb1709bd8d8e117deb9f
Sep 16 18:54:06.093546 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:06.093520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"]
Sep 16 18:54:06.095738 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:54:06.095717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c068c20_ae41_4d8e_aa2c_4fb6c9a460ab.slice/crio-74ff273b35a4c344c54ac6eaaaa0e9e6c3b5f77d8ccfcd7371288d52e9ccbe98 WatchSource:0}: Error finding container 74ff273b35a4c344c54ac6eaaaa0e9e6c3b5f77d8ccfcd7371288d52e9ccbe98: Status 404 returned error can't find the container with id 74ff273b35a4c344c54ac6eaaaa0e9e6c3b5f77d8ccfcd7371288d52e9ccbe98
Sep 16 18:54:06.611529 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:06.611492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" event={"ID":"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab","Type":"ContainerStarted","Data":"40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4"}
Sep 16 18:54:06.611716 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:06.611565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" event={"ID":"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab","Type":"ContainerStarted","Data":"74ff273b35a4c344c54ac6eaaaa0e9e6c3b5f77d8ccfcd7371288d52e9ccbe98"}
Sep 16 18:54:06.613287 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:06.613252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" event={"ID":"32135d58-9b0f-4b93-bdd7-9b0068782b3b","Type":"ContainerStarted","Data":"264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50"}
Sep 16 18:54:06.613433 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:06.613317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" event={"ID":"32135d58-9b0f-4b93-bdd7-9b0068782b3b","Type":"ContainerStarted","Data":"e27645463086ceb295e5655b8a4e695465e4aef5c8b9fb1709bd8d8e117deb9f"}
Sep 16 18:54:07.332662 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:07.332606    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="988f6672-4c35-4482-a933-3b78ac5f1c0b" path="/var/lib/kubelet/pods/988f6672-4c35-4482-a933-3b78ac5f1c0b/volumes"
Sep 16 18:54:10.635129 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:10.635094    2578 generic.go:356] "Generic (PLEG): container finished" podID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerID="264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50" exitCode=0
Sep 16 18:54:10.635573 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:10.635169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" event={"ID":"32135d58-9b0f-4b93-bdd7-9b0068782b3b","Type":"ContainerDied","Data":"264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50"}
Sep 16 18:54:10.636888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:10.636865    2578 generic.go:356] "Generic (PLEG): container finished" podID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerID="40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4" exitCode=0
Sep 16 18:54:10.637044 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:10.636928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" event={"ID":"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab","Type":"ContainerDied","Data":"40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4"}
Sep 16 18:54:11.644167 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.644121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" event={"ID":"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab","Type":"ContainerStarted","Data":"d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa"}
Sep 16 18:54:11.644634 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.644207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" event={"ID":"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab","Type":"ContainerStarted","Data":"56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5"}
Sep 16 18:54:11.644634 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.644533    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:11.644766 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.644700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:11.646402 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.646369    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:54:11.646526 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.646469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" event={"ID":"32135d58-9b0f-4b93-bdd7-9b0068782b3b","Type":"ContainerStarted","Data":"0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236"}
Sep 16 18:54:11.646572 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.646536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" event={"ID":"32135d58-9b0f-4b93-bdd7-9b0068782b3b","Type":"ContainerStarted","Data":"928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3"}
Sep 16 18:54:11.646721 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.646707    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:11.646771 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.646751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:11.647961 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.647938    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:54:11.667729 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.667688    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podStartSLOduration=2.829857281 podStartE2EDuration="6.667675304s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.837818023s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:11.666993488 +0000 UTC m=+804.797540218" watchObservedRunningTime="2026-09-16 18:54:11.667675304 +0000 UTC m=+804.798222031"
Sep 16 18:54:11.688336 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:11.688100    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podStartSLOduration=3.017527914 podStartE2EDuration="6.688081605s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670553691s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:11.687307833 +0000 UTC m=+804.817854545" watchObservedRunningTime="2026-09-16 18:54:11.688081605 +0000 UTC m=+804.818628316"
Sep 16 18:54:12.652379 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:12.652336    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:54:12.652848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:12.652449    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:54:17.657912 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:17.657884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:54:17.658317 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:17.658130    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:54:17.658402 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:17.658377    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:54:17.658865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:17.658832    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:54:25.591498 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.591457    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"]
Sep 16 18:54:25.594978 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.594955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.597529 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.597503    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-61bdc-kube-rbac-proxy-sar-config\""
Sep 16 18:54:25.597654 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.597590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-61bdc-predictor-serving-cert\""
Sep 16 18:54:25.610052 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.610030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"]
Sep 16 18:54:25.712853 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.712814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.713046 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.712876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdvm\" (UniqueName: \"kubernetes.io/projected/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-api-access-wtdvm\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.713046 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.712981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.713153 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.713053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9367fd48-e17c-4177-a4db-ec1f8971663a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.713153 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.713078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.813762 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.813715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.813762 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.813779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdvm\" (UniqueName: \"kubernetes.io/projected/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-api-access-wtdvm\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.814060 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.813815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.814060 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.813842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9367fd48-e17c-4177-a4db-ec1f8971663a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.814060 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.813865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.814060 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:54:25.813979    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-61bdc-predictor-serving-cert" not found
Sep 16 18:54:25.814278 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:54:25.814069    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls podName:9367fd48-e17c-4177-a4db-ec1f8971663a nodeName:}" failed. No retries permitted until 2026-09-16 18:54:26.314049178 +0000 UTC m=+819.444595870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" (UID: "9367fd48-e17c-4177-a4db-ec1f8971663a") : secret "isvc-sklearn-graph-raw-hpa-61bdc-predictor-serving-cert" not found
Sep 16 18:54:25.814345 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.814326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9367fd48-e17c-4177-a4db-ec1f8971663a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.814577 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.814556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.814667 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.814559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:25.825662 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:25.825634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdvm\" (UniqueName: \"kubernetes.io/projected/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-api-access-wtdvm\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:26.318338 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:26.318287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:26.320947 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:26.320919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:26.505687 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:26.505644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:26.784912 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:26.784881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"]
Sep 16 18:54:26.788965 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:54:26.788933    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9367fd48_e17c_4177_a4db_ec1f8971663a.slice/crio-85eb32bb538afd926ea8ab4123dc8fd0afa65585321bc893922a08d369976000 WatchSource:0}: Error finding container 85eb32bb538afd926ea8ab4123dc8fd0afa65585321bc893922a08d369976000: Status 404 returned error can't find the container with id 85eb32bb538afd926ea8ab4123dc8fd0afa65585321bc893922a08d369976000
Sep 16 18:54:27.659077 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:27.659041    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:54:27.659258 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:27.659041    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:54:27.715404 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:27.715370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" event={"ID":"9367fd48-e17c-4177-a4db-ec1f8971663a","Type":"ContainerStarted","Data":"ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939"}
Sep 16 18:54:27.715554 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:27.715434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" event={"ID":"9367fd48-e17c-4177-a4db-ec1f8971663a","Type":"ContainerStarted","Data":"85eb32bb538afd926ea8ab4123dc8fd0afa65585321bc893922a08d369976000"}
Sep 16 18:54:30.731452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:30.731419    2578 generic.go:356] "Generic (PLEG): container finished" podID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerID="ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939" exitCode=0
Sep 16 18:54:30.731832 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:30.731496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" event={"ID":"9367fd48-e17c-4177-a4db-ec1f8971663a","Type":"ContainerDied","Data":"ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939"}
Sep 16 18:54:31.737841 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:31.737810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" event={"ID":"9367fd48-e17c-4177-a4db-ec1f8971663a","Type":"ContainerStarted","Data":"52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57"}
Sep 16 18:54:31.738214 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:31.737867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" event={"ID":"9367fd48-e17c-4177-a4db-ec1f8971663a","Type":"ContainerStarted","Data":"d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4"}
Sep 16 18:54:31.738214 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:31.738055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:31.760500 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:31.760458    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podStartSLOduration=3.048871521 podStartE2EDuration="6.760442709s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711571188s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:31.75909001 +0000 UTC m=+824.889636721" watchObservedRunningTime="2026-09-16 18:54:31.760442709 +0000 UTC m=+824.890989421"
Sep 16 18:54:32.741967 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:32.741940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:32.743396 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:32.743368    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:54:33.746074 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:33.746042    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:54:37.659255 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:37.659211    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:54:37.659697 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:37.659207    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:54:38.750620 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:38.750594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:54:38.751113 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:38.751084    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:54:47.659362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:47.659278    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:54:47.659703 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:47.659277    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:54:48.751102 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:48.751063    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:54:57.658505 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:57.658467    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:54:57.658960 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:57.658758    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:54:58.751270 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:54:58.751233    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:55:07.659240 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:07.659206    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:07.659625 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:07.659214    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:55:08.751724 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:08.751688    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:55:17.658915 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:17.658873    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:17.659387 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:17.658873    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:55:18.751847 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:18.751804    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:55:27.659707 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:27.659680    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:55:27.660134 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:27.660116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:55:27.815040 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:27.814989    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"]
Sep 16 18:55:27.975263 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:27.975196    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container" containerID="cri-o://56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5" gracePeriod=30
Sep 16 18:55:27.975451 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:27.975219    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kube-rbac-proxy" containerID="cri-o://d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa" gracePeriod=30
Sep 16 18:55:28.751306 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:28.751264    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:55:28.981769 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:28.981737    2578 generic.go:356] "Generic (PLEG): container finished" podID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerID="d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa" exitCode=2
Sep 16 18:55:28.981936 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:28.981812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" event={"ID":"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab","Type":"ContainerDied","Data":"d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa"}
Sep 16 18:55:32.653074 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.653035    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 16 18:55:32.833307 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.833280    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:55:32.927398 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.927308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46x6z\" (UniqueName: \"kubernetes.io/projected/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-api-access-46x6z\") pod \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") "
Sep 16 18:55:32.927398 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.927355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-proxy-tls\") pod \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") "
Sep 16 18:55:32.927612 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.927403    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kserve-provision-location\") pod \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") "
Sep 16 18:55:32.927612 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.927447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-cabundle-cert\") pod \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") "
Sep 16 18:55:32.927612 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.927516    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-rbac-proxy-sar-config\") pod \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\" (UID: \"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab\") "
Sep 16 18:55:32.927752 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.927711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" (UID: "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:32.927868 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.927843    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" (UID: "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:32.927942 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.927865    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" (UID: "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:32.929706 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.929685    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" (UID: "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:32.929818 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:32.929738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-api-access-46x6z" (OuterVolumeSpecName: "kube-api-access-46x6z") pod "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" (UID: "9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab"). InnerVolumeSpecName "kube-api-access-46x6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:33.001670 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.001636    2578 generic.go:356] "Generic (PLEG): container finished" podID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerID="56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5" exitCode=0
Sep 16 18:55:33.001865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.001708    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"
Sep 16 18:55:33.001865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.001726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" event={"ID":"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab","Type":"ContainerDied","Data":"56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5"}
Sep 16 18:55:33.001865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.001788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz" event={"ID":"9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab","Type":"ContainerDied","Data":"74ff273b35a4c344c54ac6eaaaa0e9e6c3b5f77d8ccfcd7371288d52e9ccbe98"}
Sep 16 18:55:33.001865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.001814    2578 scope.go:117] "RemoveContainer" containerID="d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa"
Sep 16 18:55:33.010785 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.010764    2578 scope.go:117] "RemoveContainer" containerID="56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5"
Sep 16 18:55:33.019542 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.019518    2578 scope.go:117] "RemoveContainer" containerID="40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4"
Sep 16 18:55:33.027190 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.027165    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"]
Sep 16 18:55:33.028622 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.028600    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:33.028622 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.028623    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:33.028764 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.028634    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:33.028764 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.028643    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46x6z\" (UniqueName: \"kubernetes.io/projected/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-kube-api-access-46x6z\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:33.028764 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.028652    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:33.028995 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.028978    2578 scope.go:117] "RemoveContainer" containerID="d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa"
Sep 16 18:55:33.029309 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:55:33.029290    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa\": container with ID starting with d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa not found: ID does not exist" containerID="d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa"
Sep 16 18:55:33.029375 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.029318    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa"} err="failed to get container status \"d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa\": rpc error: code = NotFound desc = could not find container \"d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa\": container with ID starting with d9e353455a98ff8eb7639712431f32ba8a2d09292ff5ac3d08673691f8d539fa not found: ID does not exist"
Sep 16 18:55:33.029375 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.029334    2578 scope.go:117] "RemoveContainer" containerID="56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5"
Sep 16 18:55:33.029595 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:55:33.029576    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5\": container with ID starting with 56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5 not found: ID does not exist" containerID="56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5"
Sep 16 18:55:33.029679 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.029598    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5"} err="failed to get container status \"56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5\": rpc error: code = NotFound desc = could not find container \"56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5\": container with ID starting with 56d6b55e028d95ae0f2701369ce59e4afb50e990ab3862329d093ffefe17a0d5 not found: ID does not exist"
Sep 16 18:55:33.029679 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.029611    2578 scope.go:117] "RemoveContainer" containerID="40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4"
Sep 16 18:55:33.029921 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:55:33.029900    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4\": container with ID starting with 40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4 not found: ID does not exist" containerID="40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4"
Sep 16 18:55:33.030047 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.029929    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4"} err="failed to get container status \"40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4\": rpc error: code = NotFound desc = could not find container \"40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4\": container with ID starting with 40a4a6a48789076af1f9b6274980c0f0e8a70d0313a0ef638e158af33ae0d8d4 not found: ID does not exist"
Sep 16 18:55:33.032019 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.031989    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-knzqz"]
Sep 16 18:55:33.332312 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:33.332269    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" path="/var/lib/kubelet/pods/9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab/volumes"
Sep 16 18:55:35.484353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:35.484319    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"]
Sep 16 18:55:35.484766 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:35.484567    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" containerID="cri-o://928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3" gracePeriod=30
Sep 16 18:55:35.484766 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:35.484585    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kube-rbac-proxy" containerID="cri-o://0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236" gracePeriod=30
Sep 16 18:55:36.017552 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:36.017520    2578 generic.go:356] "Generic (PLEG): container finished" podID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerID="0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236" exitCode=2
Sep 16 18:55:36.017748 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:36.017552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" event={"ID":"32135d58-9b0f-4b93-bdd7-9b0068782b3b","Type":"ContainerDied","Data":"0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236"}
Sep 16 18:55:37.653029 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:37.652967    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 16 18:55:37.658783 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:37.658760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:38.751725 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:38.751683    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:55:40.330728 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.330698    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:55:40.383774 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.383694    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-rbac-proxy-sar-config\") pod \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") "
Sep 16 18:55:40.383774 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.383740    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kserve-provision-location\") pod \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") "
Sep 16 18:55:40.383949 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.383793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn5mg\" (UniqueName: \"kubernetes.io/projected/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-api-access-tn5mg\") pod \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") "
Sep 16 18:55:40.383949 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.383824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32135d58-9b0f-4b93-bdd7-9b0068782b3b-proxy-tls\") pod \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") "
Sep 16 18:55:40.383949 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.383869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-cabundle-cert\") pod \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\" (UID: \"32135d58-9b0f-4b93-bdd7-9b0068782b3b\") "
Sep 16 18:55:40.384181 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.384150    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32135d58-9b0f-4b93-bdd7-9b0068782b3b" (UID: "32135d58-9b0f-4b93-bdd7-9b0068782b3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:40.384237 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.384217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32135d58-9b0f-4b93-bdd7-9b0068782b3b" (UID: "32135d58-9b0f-4b93-bdd7-9b0068782b3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:40.384286 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.384267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32135d58-9b0f-4b93-bdd7-9b0068782b3b" (UID: "32135d58-9b0f-4b93-bdd7-9b0068782b3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:40.386121 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.386099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-api-access-tn5mg" (OuterVolumeSpecName: "kube-api-access-tn5mg") pod "32135d58-9b0f-4b93-bdd7-9b0068782b3b" (UID: "32135d58-9b0f-4b93-bdd7-9b0068782b3b"). InnerVolumeSpecName "kube-api-access-tn5mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:40.386121 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.386101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32135d58-9b0f-4b93-bdd7-9b0068782b3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32135d58-9b0f-4b93-bdd7-9b0068782b3b" (UID: "32135d58-9b0f-4b93-bdd7-9b0068782b3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:40.484669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.484635    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32135d58-9b0f-4b93-bdd7-9b0068782b3b-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:40.484669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.484666    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:40.484669 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.484675    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:40.484906 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.484692    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:40.484906 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:40.484701    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn5mg\" (UniqueName: \"kubernetes.io/projected/32135d58-9b0f-4b93-bdd7-9b0068782b3b-kube-api-access-tn5mg\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.040690 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.040646    2578 generic.go:356] "Generic (PLEG): container finished" podID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerID="928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3" exitCode=0
Sep 16 18:55:41.040836 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.040701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" event={"ID":"32135d58-9b0f-4b93-bdd7-9b0068782b3b","Type":"ContainerDied","Data":"928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3"}
Sep 16 18:55:41.040836 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.040719    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"
Sep 16 18:55:41.040836 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.040770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz" event={"ID":"32135d58-9b0f-4b93-bdd7-9b0068782b3b","Type":"ContainerDied","Data":"e27645463086ceb295e5655b8a4e695465e4aef5c8b9fb1709bd8d8e117deb9f"}
Sep 16 18:55:41.040836 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.040809    2578 scope.go:117] "RemoveContainer" containerID="0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236"
Sep 16 18:55:41.049356 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.049334    2578 scope.go:117] "RemoveContainer" containerID="928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3"
Sep 16 18:55:41.056667 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.056652    2578 scope.go:117] "RemoveContainer" containerID="264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50"
Sep 16 18:55:41.063775 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.063752    2578 scope.go:117] "RemoveContainer" containerID="0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236"
Sep 16 18:55:41.064065 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:55:41.064044    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236\": container with ID starting with 0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236 not found: ID does not exist" containerID="0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236"
Sep 16 18:55:41.064136 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.064073    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236"} err="failed to get container status \"0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236\": rpc error: code = NotFound desc = could not find container \"0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236\": container with ID starting with 0ac89fa122dcc1d7ea6658cf5de54609ea665fda8b7264774f520d0d0c75e236 not found: ID does not exist"
Sep 16 18:55:41.064136 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.064090    2578 scope.go:117] "RemoveContainer" containerID="928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3"
Sep 16 18:55:41.064331 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:55:41.064314    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3\": container with ID starting with 928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3 not found: ID does not exist" containerID="928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3"
Sep 16 18:55:41.064370 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.064337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3"} err="failed to get container status \"928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3\": rpc error: code = NotFound desc = could not find container \"928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3\": container with ID starting with 928ae17cf2b691a402cfc56535f116d00f2668705a5d2bd3e24c537ac614c8f3 not found: ID does not exist"
Sep 16 18:55:41.064370 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.064364    2578 scope.go:117] "RemoveContainer" containerID="264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50"
Sep 16 18:55:41.064609 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:55:41.064579    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50\": container with ID starting with 264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50 not found: ID does not exist" containerID="264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50"
Sep 16 18:55:41.064660 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.064614    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50"} err="failed to get container status \"264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50\": rpc error: code = NotFound desc = could not find container \"264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50\": container with ID starting with 264445edc51d9824f4e28ff40937d1a19390e9e1cce002c712e997ebe5edca50 not found: ID does not exist"
Sep 16 18:55:41.071527 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.071506    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"]
Sep 16 18:55:41.077592 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.077570    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-h74zz"]
Sep 16 18:55:41.331856 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:41.331772    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" path="/var/lib/kubelet/pods/32135d58-9b0f-4b93-bdd7-9b0068782b3b/volumes"
Sep 16 18:55:45.758172 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758137    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"]
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758418    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="storage-initializer"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="storage-initializer"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758438    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758444    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758452    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kube-rbac-proxy"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758458    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kube-rbac-proxy"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758473    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="storage-initializer"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758478    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="storage-initializer"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758485    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kube-rbac-proxy"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758490    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kube-rbac-proxy"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758498    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container"
Sep 16 18:55:45.758545 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758504    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container"
Sep 16 18:55:45.758888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758553    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kserve-container"
Sep 16 18:55:45.758888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758560    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kube-rbac-proxy"
Sep 16 18:55:45.758888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758565    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="32135d58-9b0f-4b93-bdd7-9b0068782b3b" containerName="kserve-container"
Sep 16 18:55:45.758888 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.758573    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c068c20-ae41-4d8e-aa2c-4fb6c9a460ab" containerName="kube-rbac-proxy"
Sep 16 18:55:45.761500 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.761482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.764211 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.764185    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-7f5d46-kube-rbac-proxy-sar-config\""
Sep 16 18:55:45.764317 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.764222    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-7f5d46-predictor-serving-cert\""
Sep 16 18:55:45.772181 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.772157    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"]
Sep 16 18:55:45.824896 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.824856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-cabundle-cert\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.824896 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.824901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8695aa-9809-481a-a42a-23e11cba2349-kserve-provision-location\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.825171 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.824929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.825171 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.824983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zjn\" (UniqueName: \"kubernetes.io/projected/6e8695aa-9809-481a-a42a-23e11cba2349-kube-api-access-f2zjn\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.825171 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.825040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-kube-rbac-proxy-sar-config\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.925734 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.925690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-cabundle-cert\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.925734 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.925738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8695aa-9809-481a-a42a-23e11cba2349-kserve-provision-location\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.926044 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.925768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.926044 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.925795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zjn\" (UniqueName: \"kubernetes.io/projected/6e8695aa-9809-481a-a42a-23e11cba2349-kube-api-access-f2zjn\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.926044 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.925851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-kube-rbac-proxy-sar-config\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.926044 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:55:45.925900    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-serving-cert: secret "isvc-primary-7f5d46-predictor-serving-cert" not found
Sep 16 18:55:45.926044 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:55:45.925988    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls podName:6e8695aa-9809-481a-a42a-23e11cba2349 nodeName:}" failed. No retries permitted until 2026-09-16 18:55:46.425966905 +0000 UTC m=+899.556513593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls") pod "isvc-primary-7f5d46-predictor-58798f9794-tdr4k" (UID: "6e8695aa-9809-481a-a42a-23e11cba2349") : secret "isvc-primary-7f5d46-predictor-serving-cert" not found
Sep 16 18:55:45.926319 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.926194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8695aa-9809-481a-a42a-23e11cba2349-kserve-provision-location\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.926472 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.926453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-kube-rbac-proxy-sar-config\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.926589 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.926569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-cabundle-cert\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:45.937473 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:45.937440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zjn\" (UniqueName: \"kubernetes.io/projected/6e8695aa-9809-481a-a42a-23e11cba2349-kube-api-access-f2zjn\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:46.430276 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:46.430240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:46.432961 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:46.432921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls\") pod \"isvc-primary-7f5d46-predictor-58798f9794-tdr4k\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") " pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:46.672805 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:46.672766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:46.948563 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:46.948536    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"]
Sep 16 18:55:46.950855 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:55:46.950829    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8695aa_9809_481a_a42a_23e11cba2349.slice/crio-946a1e594417dbad1b113ceb23734c831514d3970ba2068c08eef97d62f67ab9 WatchSource:0}: Error finding container 946a1e594417dbad1b113ceb23734c831514d3970ba2068c08eef97d62f67ab9: Status 404 returned error can't find the container with id 946a1e594417dbad1b113ceb23734c831514d3970ba2068c08eef97d62f67ab9
Sep 16 18:55:47.067418 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:47.067383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" event={"ID":"6e8695aa-9809-481a-a42a-23e11cba2349","Type":"ContainerStarted","Data":"d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71"}
Sep 16 18:55:47.067578 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:47.067446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" event={"ID":"6e8695aa-9809-481a-a42a-23e11cba2349","Type":"ContainerStarted","Data":"946a1e594417dbad1b113ceb23734c831514d3970ba2068c08eef97d62f67ab9"}
Sep 16 18:55:47.273318 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:47.273285    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:55:47.276764 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:47.276743    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 18:55:48.752208 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:48.752179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:55:51.084568 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:51.084537    2578 generic.go:356] "Generic (PLEG): container finished" podID="6e8695aa-9809-481a-a42a-23e11cba2349" containerID="d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71" exitCode=0
Sep 16 18:55:51.084951 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:51.084618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" event={"ID":"6e8695aa-9809-481a-a42a-23e11cba2349","Type":"ContainerDied","Data":"d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71"}
Sep 16 18:55:52.091463 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:52.091426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" event={"ID":"6e8695aa-9809-481a-a42a-23e11cba2349","Type":"ContainerStarted","Data":"c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da"}
Sep 16 18:55:52.091828 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:52.091489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" event={"ID":"6e8695aa-9809-481a-a42a-23e11cba2349","Type":"ContainerStarted","Data":"3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2"}
Sep 16 18:55:52.091828 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:52.091678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:52.113250 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:52.113204    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podStartSLOduration=3.243444164 podStartE2EDuration="7.113187864s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8697437s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:52.111724805 +0000 UTC m=+905.242271528" watchObservedRunningTime="2026-09-16 18:55:52.113187864 +0000 UTC m=+905.243734572"
Sep 16 18:55:53.095660 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:53.095633    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:53.097101 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:53.097069    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:55:54.100095 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:54.100060    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:55:59.104814 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:59.104783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:55:59.105283 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:55:59.105256    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:56:09.106203 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:09.106167    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:56:15.863724 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:15.863690    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"]
Sep 16 18:56:15.864114 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:15.863960    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" containerID="cri-o://d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4" gracePeriod=30
Sep 16 18:56:15.864114 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:15.864025    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kube-rbac-proxy" containerID="cri-o://52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57" gracePeriod=30
Sep 16 18:56:15.932024 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:15.931977    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"]
Sep 16 18:56:15.935324 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:15.935304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:15.939048 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:15.939024    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-f5e4d-predictor-serving-cert\""
Sep 16 18:56:15.939163 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:15.939026    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-f5e4d-kube-rbac-proxy-sar-config\""
Sep 16 18:56:15.946846 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:15.946825    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"]
Sep 16 18:56:16.075269 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.075239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dn2\" (UniqueName: \"kubernetes.io/projected/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-api-access-89dn2\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.075409 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.075274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.075409 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.075305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cad9dccd-7262-41d4-97bd-051ad5f5e631-proxy-tls\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.176188 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.176121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dn2\" (UniqueName: \"kubernetes.io/projected/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-api-access-89dn2\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.176188 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.176158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.176349 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.176197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cad9dccd-7262-41d4-97bd-051ad5f5e631-proxy-tls\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.176781 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.176761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.178707 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.178689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cad9dccd-7262-41d4-97bd-051ad5f5e631-proxy-tls\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.187257 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.187234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dn2\" (UniqueName: \"kubernetes.io/projected/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-api-access-89dn2\") pod \"message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") " pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.191871 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.191848    2578 generic.go:356] "Generic (PLEG): container finished" podID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerID="52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57" exitCode=2
Sep 16 18:56:16.191977 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.191917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" event={"ID":"9367fd48-e17c-4177-a4db-ec1f8971663a","Type":"ContainerDied","Data":"52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57"}
Sep 16 18:56:16.245249 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.245220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:16.517624 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.517596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"]
Sep 16 18:56:16.520036 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:56:16.519987    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad9dccd_7262_41d4_97bd_051ad5f5e631.slice/crio-f74437d10d2cab9cd4c9b5f0ff5c2051ace71f87e8d4f1247c5816f19b3fb223 WatchSource:0}: Error finding container f74437d10d2cab9cd4c9b5f0ff5c2051ace71f87e8d4f1247c5816f19b3fb223: Status 404 returned error can't find the container with id f74437d10d2cab9cd4c9b5f0ff5c2051ace71f87e8d4f1247c5816f19b3fb223
Sep 16 18:56:16.522335 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:16.522316    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:56:17.198826 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:17.198797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" event={"ID":"cad9dccd-7262-41d4-97bd-051ad5f5e631","Type":"ContainerStarted","Data":"f74437d10d2cab9cd4c9b5f0ff5c2051ace71f87e8d4f1247c5816f19b3fb223"}
Sep 16 18:56:18.204573 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:18.204540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" event={"ID":"cad9dccd-7262-41d4-97bd-051ad5f5e631","Type":"ContainerStarted","Data":"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3"}
Sep 16 18:56:18.205039 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:18.204597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" event={"ID":"cad9dccd-7262-41d4-97bd-051ad5f5e631","Type":"ContainerStarted","Data":"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e"}
Sep 16 18:56:18.205039 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:18.204622    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:18.205039 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:18.204672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:18.206511 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:18.206491    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:18.240206 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:18.240166    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" podStartSLOduration=2.050953796 podStartE2EDuration="3.240150863s" totalImagesPullingTime="1.189197067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:18.239041903 +0000 UTC m=+931.369588613" watchObservedRunningTime="2026-09-16 18:56:18.240150863 +0000 UTC m=+931.370697573"
Sep 16 18:56:18.746658 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:18.746610    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 16 18:56:18.751964 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:18.751939    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:56:19.105918 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:19.105827    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:56:20.504316 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.504293    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:56:20.613071 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.612964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtdvm\" (UniqueName: \"kubernetes.io/projected/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-api-access-wtdvm\") pod \"9367fd48-e17c-4177-a4db-ec1f8971663a\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") "
Sep 16 18:56:20.613071 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.613020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-rbac-proxy-sar-config\") pod \"9367fd48-e17c-4177-a4db-ec1f8971663a\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") "
Sep 16 18:56:20.613071 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.613039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls\") pod \"9367fd48-e17c-4177-a4db-ec1f8971663a\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") "
Sep 16 18:56:20.613071 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.613062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-cabundle-cert\") pod \"9367fd48-e17c-4177-a4db-ec1f8971663a\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") "
Sep 16 18:56:20.613389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.613116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9367fd48-e17c-4177-a4db-ec1f8971663a-kserve-provision-location\") pod \"9367fd48-e17c-4177-a4db-ec1f8971663a\" (UID: \"9367fd48-e17c-4177-a4db-ec1f8971663a\") "
Sep 16 18:56:20.613467 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.613441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9367fd48-e17c-4177-a4db-ec1f8971663a" (UID: "9367fd48-e17c-4177-a4db-ec1f8971663a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:20.613505 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.613453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9367fd48-e17c-4177-a4db-ec1f8971663a" (UID: "9367fd48-e17c-4177-a4db-ec1f8971663a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:20.613505 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.613461    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9367fd48-e17c-4177-a4db-ec1f8971663a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9367fd48-e17c-4177-a4db-ec1f8971663a" (UID: "9367fd48-e17c-4177-a4db-ec1f8971663a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:20.615387 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.615356    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9367fd48-e17c-4177-a4db-ec1f8971663a" (UID: "9367fd48-e17c-4177-a4db-ec1f8971663a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:20.615502 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.615438    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-api-access-wtdvm" (OuterVolumeSpecName: "kube-api-access-wtdvm") pod "9367fd48-e17c-4177-a4db-ec1f8971663a" (UID: "9367fd48-e17c-4177-a4db-ec1f8971663a"). InnerVolumeSpecName "kube-api-access-wtdvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:20.713877 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.713828    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtdvm\" (UniqueName: \"kubernetes.io/projected/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-api-access-wtdvm\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:56:20.713877 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.713871    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:56:20.713877 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.713883    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9367fd48-e17c-4177-a4db-ec1f8971663a-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:56:20.713877 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.713892    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9367fd48-e17c-4177-a4db-ec1f8971663a-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:56:20.714208 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:20.713902    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9367fd48-e17c-4177-a4db-ec1f8971663a-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:56:21.218930 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.218895    2578 generic.go:356] "Generic (PLEG): container finished" podID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerID="d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4" exitCode=0
Sep 16 18:56:21.219150 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.218955    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"
Sep 16 18:56:21.219150 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.218975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" event={"ID":"9367fd48-e17c-4177-a4db-ec1f8971663a","Type":"ContainerDied","Data":"d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4"}
Sep 16 18:56:21.219150 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.219058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l" event={"ID":"9367fd48-e17c-4177-a4db-ec1f8971663a","Type":"ContainerDied","Data":"85eb32bb538afd926ea8ab4123dc8fd0afa65585321bc893922a08d369976000"}
Sep 16 18:56:21.219150 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.219095    2578 scope.go:117] "RemoveContainer" containerID="52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57"
Sep 16 18:56:21.228035 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.227995    2578 scope.go:117] "RemoveContainer" containerID="d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4"
Sep 16 18:56:21.235637 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.235620    2578 scope.go:117] "RemoveContainer" containerID="ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939"
Sep 16 18:56:21.243416 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.243388    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"]
Sep 16 18:56:21.243945 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.243929    2578 scope.go:117] "RemoveContainer" containerID="52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57"
Sep 16 18:56:21.244242 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:56:21.244223    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57\": container with ID starting with 52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57 not found: ID does not exist" containerID="52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57"
Sep 16 18:56:21.244316 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.244250    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57"} err="failed to get container status \"52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57\": rpc error: code = NotFound desc = could not find container \"52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57\": container with ID starting with 52f91d8bbe23f56799802c4a5051bcee3e6de0c82ad8c82ac28df4b59b0ddf57 not found: ID does not exist"
Sep 16 18:56:21.244316 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.244267    2578 scope.go:117] "RemoveContainer" containerID="d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4"
Sep 16 18:56:21.244555 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:56:21.244537    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4\": container with ID starting with d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4 not found: ID does not exist" containerID="d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4"
Sep 16 18:56:21.244595 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.244561    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4"} err="failed to get container status \"d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4\": rpc error: code = NotFound desc = could not find container \"d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4\": container with ID starting with d5c479bb8bc6d24388047ea138f68419aa5cf6bbfafc8cd99ccffe9635eb95d4 not found: ID does not exist"
Sep 16 18:56:21.244595 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.244577    2578 scope.go:117] "RemoveContainer" containerID="ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939"
Sep 16 18:56:21.244846 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:56:21.244830    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939\": container with ID starting with ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939 not found: ID does not exist" containerID="ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939"
Sep 16 18:56:21.244897 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.244850    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939"} err="failed to get container status \"ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939\": rpc error: code = NotFound desc = could not find container \"ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939\": container with ID starting with ae82d36c2c4661f1af57d64aa69bf7f8f7673cc14f31e7d9f81f079b76990939 not found: ID does not exist"
Sep 16 18:56:21.248043 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.248023    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-61bdc-predictor-686569965f-9w74l"]
Sep 16 18:56:21.331149 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:21.331109    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" path="/var/lib/kubelet/pods/9367fd48-e17c-4177-a4db-ec1f8971663a/volumes"
Sep 16 18:56:25.217461 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.217429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:56:25.974040 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.973993    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"]
Sep 16 18:56:25.974346 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.974335    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="storage-initializer"
Sep 16 18:56:25.974389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.974348    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="storage-initializer"
Sep 16 18:56:25.974389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.974358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kube-rbac-proxy"
Sep 16 18:56:25.974389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.974363    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kube-rbac-proxy"
Sep 16 18:56:25.974389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.974372    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container"
Sep 16 18:56:25.974389 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.974381    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container"
Sep 16 18:56:25.974541 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.974437    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kube-rbac-proxy"
Sep 16 18:56:25.974541 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.974445    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9367fd48-e17c-4177-a4db-ec1f8971663a" containerName="kserve-container"
Sep 16 18:56:25.977809 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.977793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:25.980494 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.980468    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-f5e4d-predictor-serving-cert\""
Sep 16 18:56:25.980980 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.980966    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-f5e4d-kube-rbac-proxy-sar-config\""
Sep 16 18:56:25.996495 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:25.996466    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"]
Sep 16 18:56:26.152805 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.152722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5h77\" (UniqueName: \"kubernetes.io/projected/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-api-access-v5h77\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.152805 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.152772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-proxy-tls\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.152805 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.152795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-cabundle-cert\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.153024 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.152826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.153024 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.152866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kserve-provision-location\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.253608 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.253565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-proxy-tls\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.253608 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.253612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-cabundle-cert\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.254177 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.253740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.254177 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.253804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kserve-provision-location\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.254177 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.253844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5h77\" (UniqueName: \"kubernetes.io/projected/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-api-access-v5h77\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.254342 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.254202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kserve-provision-location\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.254400 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.254379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.254488 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.254440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-cabundle-cert\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.256311 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.256293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-proxy-tls\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.263580 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.263554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5h77\" (UniqueName: \"kubernetes.io/projected/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-api-access-v5h77\") pod \"isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.290505 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.290463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:26.569390 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:26.569370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"]
Sep 16 18:56:26.572022 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:56:26.571980    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa10fe58_364e_4036_a5f3_ce0b7bfb6a15.slice/crio-c2680138bf3d8c155dd5583b51bcfe5479a6ae31f7ec3b1b774ba4c8337ff123 WatchSource:0}: Error finding container c2680138bf3d8c155dd5583b51bcfe5479a6ae31f7ec3b1b774ba4c8337ff123: Status 404 returned error can't find the container with id c2680138bf3d8c155dd5583b51bcfe5479a6ae31f7ec3b1b774ba4c8337ff123
Sep 16 18:56:27.246283 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:27.246248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerStarted","Data":"76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f"}
Sep 16 18:56:27.246463 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:27.246326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerStarted","Data":"c2680138bf3d8c155dd5583b51bcfe5479a6ae31f7ec3b1b774ba4c8337ff123"}
Sep 16 18:56:29.105439 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:29.105401    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:56:31.263797 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:31.263765    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerID="76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f" exitCode=0
Sep 16 18:56:31.264196 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:31.263840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerDied","Data":"76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f"}
Sep 16 18:56:32.271686 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:32.271651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerStarted","Data":"b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a"}
Sep 16 18:56:32.272089 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:32.271716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerStarted","Data":"2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0"}
Sep 16 18:56:32.272089 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:32.271739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerStarted","Data":"8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537"}
Sep 16 18:56:32.272089 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:32.272058    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:32.272259 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:32.272126    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:32.273498 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:32.273474    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:32.295205 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:32.295161    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podStartSLOduration=3.454125909 podStartE2EDuration="7.295148641s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841022732s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:32.294243602 +0000 UTC m=+945.424790313" watchObservedRunningTime="2026-09-16 18:56:32.295148641 +0000 UTC m=+945.425695403"
Sep 16 18:56:33.276469 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:33.276426    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:33.276953 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:33.276599    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:33.277809 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:33.277779    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:34.280172 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:34.280131    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:34.280696 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:34.280659    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:39.105457 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:39.105420    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:56:39.284809 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:39.284412    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:56:39.285126 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:39.285099    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:39.285825 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:39.285795    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:49.105954 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:49.105915    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:56:49.285915 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:49.285873    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:49.286362 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:49.286331    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:59.106215 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:59.106171    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:56:59.285929 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:59.285880    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:59.286363 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:56:59.286339    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:09.106765 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:09.106670    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:57:09.286114 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:09.286064    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:09.286525 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:09.286500    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:19.285443 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:19.285404    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:19.285943 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:19.285830    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:29.285151 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:29.285108    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:29.285569 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:29.285514    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:39.285231 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:39.285185    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:39.285690 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:39.285646    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:49.052090 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:49.052060    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"]
Sep 16 18:57:49.052491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:49.052328    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container" containerID="cri-o://3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2" gracePeriod=30
Sep 16 18:57:49.052491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:49.052346    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kube-rbac-proxy" containerID="cri-o://c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da" gracePeriod=30
Sep 16 18:57:49.101049 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:49.100974    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 16 18:57:49.286434 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:49.286407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:57:49.286865 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:49.286846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:57:49.603714 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:49.603682    2578 generic.go:356] "Generic (PLEG): container finished" podID="6e8695aa-9809-481a-a42a-23e11cba2349" containerID="c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da" exitCode=2
Sep 16 18:57:49.603878 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:49.603755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" event={"ID":"6e8695aa-9809-481a-a42a-23e11cba2349","Type":"ContainerDied","Data":"c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da"}
Sep 16 18:57:53.900464 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.900441    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:57:53.928247 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.928218    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2zjn\" (UniqueName: \"kubernetes.io/projected/6e8695aa-9809-481a-a42a-23e11cba2349-kube-api-access-f2zjn\") pod \"6e8695aa-9809-481a-a42a-23e11cba2349\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") "
Sep 16 18:57:53.928390 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.928300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8695aa-9809-481a-a42a-23e11cba2349-kserve-provision-location\") pod \"6e8695aa-9809-481a-a42a-23e11cba2349\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") "
Sep 16 18:57:53.928390 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.928332    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-kube-rbac-proxy-sar-config\") pod \"6e8695aa-9809-481a-a42a-23e11cba2349\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") "
Sep 16 18:57:53.928510 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.928388    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-cabundle-cert\") pod \"6e8695aa-9809-481a-a42a-23e11cba2349\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") "
Sep 16 18:57:53.928510 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.928427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls\") pod \"6e8695aa-9809-481a-a42a-23e11cba2349\" (UID: \"6e8695aa-9809-481a-a42a-23e11cba2349\") "
Sep 16 18:57:53.928668 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.928635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8695aa-9809-481a-a42a-23e11cba2349-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e8695aa-9809-481a-a42a-23e11cba2349" (UID: "6e8695aa-9809-481a-a42a-23e11cba2349"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:53.928763 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.928707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e8695aa-9809-481a-a42a-23e11cba2349" (UID: "6e8695aa-9809-481a-a42a-23e11cba2349"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:53.928833 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.928751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e8695aa-9809-481a-a42a-23e11cba2349" (UID: "6e8695aa-9809-481a-a42a-23e11cba2349"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:53.930580 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.930555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e8695aa-9809-481a-a42a-23e11cba2349" (UID: "6e8695aa-9809-481a-a42a-23e11cba2349"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:53.930685 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:53.930672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8695aa-9809-481a-a42a-23e11cba2349-kube-api-access-f2zjn" (OuterVolumeSpecName: "kube-api-access-f2zjn") pod "6e8695aa-9809-481a-a42a-23e11cba2349" (UID: "6e8695aa-9809-481a-a42a-23e11cba2349"). InnerVolumeSpecName "kube-api-access-f2zjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:54.029105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.029077    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zjn\" (UniqueName: \"kubernetes.io/projected/6e8695aa-9809-481a-a42a-23e11cba2349-kube-api-access-f2zjn\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:57:54.029105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.029101    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e8695aa-9809-481a-a42a-23e11cba2349-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:57:54.029105 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.029111    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:57:54.029314 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.029119    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e8695aa-9809-481a-a42a-23e11cba2349-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:57:54.029314 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.029129    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e8695aa-9809-481a-a42a-23e11cba2349-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:57:54.628610 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.628576    2578 generic.go:356] "Generic (PLEG): container finished" podID="6e8695aa-9809-481a-a42a-23e11cba2349" containerID="3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2" exitCode=0
Sep 16 18:57:54.628813 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.628658    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"
Sep 16 18:57:54.628813 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.628658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" event={"ID":"6e8695aa-9809-481a-a42a-23e11cba2349","Type":"ContainerDied","Data":"3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2"}
Sep 16 18:57:54.628813 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.628730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k" event={"ID":"6e8695aa-9809-481a-a42a-23e11cba2349","Type":"ContainerDied","Data":"946a1e594417dbad1b113ceb23734c831514d3970ba2068c08eef97d62f67ab9"}
Sep 16 18:57:54.628813 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.628761    2578 scope.go:117] "RemoveContainer" containerID="c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da"
Sep 16 18:57:54.637412 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.637395    2578 scope.go:117] "RemoveContainer" containerID="3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2"
Sep 16 18:57:54.644971 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.644954    2578 scope.go:117] "RemoveContainer" containerID="d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71"
Sep 16 18:57:54.652078 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.652064    2578 scope.go:117] "RemoveContainer" containerID="c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da"
Sep 16 18:57:54.652341 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:57:54.652320    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da\": container with ID starting with c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da not found: ID does not exist" containerID="c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da"
Sep 16 18:57:54.652416 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.652354    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da"} err="failed to get container status \"c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da\": rpc error: code = NotFound desc = could not find container \"c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da\": container with ID starting with c24985d3bb398fd5142dc95f5860059346248c7a2fd2478678961f4546a1c3da not found: ID does not exist"
Sep 16 18:57:54.652416 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.652378    2578 scope.go:117] "RemoveContainer" containerID="3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2"
Sep 16 18:57:54.652651 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:57:54.652630    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2\": container with ID starting with 3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2 not found: ID does not exist" containerID="3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2"
Sep 16 18:57:54.652706 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.652662    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2"} err="failed to get container status \"3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2\": rpc error: code = NotFound desc = could not find container \"3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2\": container with ID starting with 3dee080a359cfadfccc7467d29da1e8935c82b04278ea83ea170ac6ae20c9fb2 not found: ID does not exist"
Sep 16 18:57:54.652706 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.652689    2578 scope.go:117] "RemoveContainer" containerID="d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71"
Sep 16 18:57:54.652948 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:57:54.652931    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71\": container with ID starting with d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71 not found: ID does not exist" containerID="d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71"
Sep 16 18:57:54.653025 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.652957    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71"} err="failed to get container status \"d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71\": rpc error: code = NotFound desc = could not find container \"d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71\": container with ID starting with d9cb38705658cb2bb70b9528b9deab56e3795a9dd810b587aa5efb9f4e57df71 not found: ID does not exist"
Sep 16 18:57:54.656452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.656431    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"]
Sep 16 18:57:54.663368 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:54.663348    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-7f5d46-predictor-58798f9794-tdr4k"]
Sep 16 18:57:55.331961 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:57:55.331914    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" path="/var/lib/kubelet/pods/6e8695aa-9809-481a-a42a-23e11cba2349/volumes"
Sep 16 18:58:00.993027 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:00.992988    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f_cad9dccd-7262-41d4-97bd-051ad5f5e631/kserve-container/0.log"
Sep 16 18:58:01.167577 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.167544    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"]
Sep 16 18:58:01.167877 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.167853    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" containerID="cri-o://8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537" gracePeriod=30
Sep 16 18:58:01.167954 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.167914    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy" containerID="cri-o://2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0" gracePeriod=30
Sep 16 18:58:01.168032 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.167940    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" containerID="cri-o://b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a" gracePeriod=30
Sep 16 18:58:01.227452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.227420    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"]
Sep 16 18:58:01.227683 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.227647    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerName="kserve-container" containerID="cri-o://841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e" gracePeriod=30
Sep 16 18:58:01.227741 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.227693    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerName="kube-rbac-proxy" containerID="cri-o://bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3" gracePeriod=30
Sep 16 18:58:01.475127 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.475103    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:58:01.593987 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.593884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-rbac-proxy-sar-config\") pod \"cad9dccd-7262-41d4-97bd-051ad5f5e631\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") "
Sep 16 18:58:01.593987 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.593962    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cad9dccd-7262-41d4-97bd-051ad5f5e631-proxy-tls\") pod \"cad9dccd-7262-41d4-97bd-051ad5f5e631\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") "
Sep 16 18:58:01.593987 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.593990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89dn2\" (UniqueName: \"kubernetes.io/projected/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-api-access-89dn2\") pod \"cad9dccd-7262-41d4-97bd-051ad5f5e631\" (UID: \"cad9dccd-7262-41d4-97bd-051ad5f5e631\") "
Sep 16 18:58:01.594324 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.594292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cad9dccd-7262-41d4-97bd-051ad5f5e631" (UID: "cad9dccd-7262-41d4-97bd-051ad5f5e631"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:01.596292 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.596271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-api-access-89dn2" (OuterVolumeSpecName: "kube-api-access-89dn2") pod "cad9dccd-7262-41d4-97bd-051ad5f5e631" (UID: "cad9dccd-7262-41d4-97bd-051ad5f5e631"). InnerVolumeSpecName "kube-api-access-89dn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:01.596292 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.596278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad9dccd-7262-41d4-97bd-051ad5f5e631-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cad9dccd-7262-41d4-97bd-051ad5f5e631" (UID: "cad9dccd-7262-41d4-97bd-051ad5f5e631"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:01.665042 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.664993    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerID="2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0" exitCode=2
Sep 16 18:58:01.665203 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.665074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerDied","Data":"2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0"}
Sep 16 18:58:01.666510 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.666490    2578 generic.go:356] "Generic (PLEG): container finished" podID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerID="bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3" exitCode=2
Sep 16 18:58:01.666510 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.666507    2578 generic.go:356] "Generic (PLEG): container finished" podID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerID="841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e" exitCode=2
Sep 16 18:58:01.666655 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.666541    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"
Sep 16 18:58:01.666655 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.666575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" event={"ID":"cad9dccd-7262-41d4-97bd-051ad5f5e631","Type":"ContainerDied","Data":"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3"}
Sep 16 18:58:01.666655 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.666632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" event={"ID":"cad9dccd-7262-41d4-97bd-051ad5f5e631","Type":"ContainerDied","Data":"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e"}
Sep 16 18:58:01.666790 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.666666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f" event={"ID":"cad9dccd-7262-41d4-97bd-051ad5f5e631","Type":"ContainerDied","Data":"f74437d10d2cab9cd4c9b5f0ff5c2051ace71f87e8d4f1247c5816f19b3fb223"}
Sep 16 18:58:01.666790 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.666707    2578 scope.go:117] "RemoveContainer" containerID="bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3"
Sep 16 18:58:01.674752 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.674730    2578 scope.go:117] "RemoveContainer" containerID="841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e"
Sep 16 18:58:01.681996 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.681980    2578 scope.go:117] "RemoveContainer" containerID="bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3"
Sep 16 18:58:01.682294 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:58:01.682275    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3\": container with ID starting with bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3 not found: ID does not exist" containerID="bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3"
Sep 16 18:58:01.682352 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.682303    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3"} err="failed to get container status \"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3\": rpc error: code = NotFound desc = could not find container \"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3\": container with ID starting with bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3 not found: ID does not exist"
Sep 16 18:58:01.682352 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.682319    2578 scope.go:117] "RemoveContainer" containerID="841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e"
Sep 16 18:58:01.682564 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:58:01.682547    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e\": container with ID starting with 841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e not found: ID does not exist" containerID="841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e"
Sep 16 18:58:01.682625 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.682572    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e"} err="failed to get container status \"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e\": rpc error: code = NotFound desc = could not find container \"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e\": container with ID starting with 841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e not found: ID does not exist"
Sep 16 18:58:01.682625 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.682593    2578 scope.go:117] "RemoveContainer" containerID="bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3"
Sep 16 18:58:01.682837 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.682821    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3"} err="failed to get container status \"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3\": rpc error: code = NotFound desc = could not find container \"bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3\": container with ID starting with bb461db717457f3b0db268cd7cfadb188f367a3a51920ea4d90f59d5d166bef3 not found: ID does not exist"
Sep 16 18:58:01.682883 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.682838    2578 scope.go:117] "RemoveContainer" containerID="841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e"
Sep 16 18:58:01.683077 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.683050    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e"} err="failed to get container status \"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e\": rpc error: code = NotFound desc = could not find container \"841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e\": container with ID starting with 841cdcaea173d6da42ac9f48378ff17dd8a2854b921358bb524653f3f6ae300e not found: ID does not exist"
Sep 16 18:58:01.694573 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.694549    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:58:01.694692 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.694574    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cad9dccd-7262-41d4-97bd-051ad5f5e631-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:58:01.694692 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.694589    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dn2\" (UniqueName: \"kubernetes.io/projected/cad9dccd-7262-41d4-97bd-051ad5f5e631-kube-api-access-89dn2\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:58:01.696711 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.696691    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"]
Sep 16 18:58:01.702392 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:01.702372    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f5e4d-predictor-584bb688b4-hz69f"]
Sep 16 18:58:03.331339 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:03.331301    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" path="/var/lib/kubelet/pods/cad9dccd-7262-41d4-97bd-051ad5f5e631/volumes"
Sep 16 18:58:04.281423 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:04.281388    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:06.690838 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:06.690798    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerID="8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537" exitCode=0
Sep 16 18:58:06.691222 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:06.690891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerDied","Data":"8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537"}
Sep 16 18:58:09.280569 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.280524    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:09.285585 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.285546    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:09.287828 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.287803    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:58:09.500921 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.500888    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"]
Sep 16 18:58:09.501217 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501202    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kube-rbac-proxy"
Sep 16 18:58:09.501217 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501217    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kube-rbac-proxy"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501233    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501238    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501245    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerName="kube-rbac-proxy"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501251    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerName="kube-rbac-proxy"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501261    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="storage-initializer"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501266    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="storage-initializer"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501277    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerName="kserve-container"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501282    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerName="kserve-container"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501328    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kube-rbac-proxy"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501338    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerName="kube-rbac-proxy"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501344    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad9dccd-7262-41d4-97bd-051ad5f5e631" containerName="kserve-container"
Sep 16 18:58:09.501353 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.501351    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8695aa-9809-481a-a42a-23e11cba2349" containerName="kserve-container"
Sep 16 18:58:09.504801 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.504775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.507369 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.507350    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fded7-kube-rbac-proxy-sar-config\""
Sep 16 18:58:09.507470 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.507353    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fded7-predictor-serving-cert\""
Sep 16 18:58:09.515935 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.515913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"]
Sep 16 18:58:09.557205 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.557135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.557205 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.557169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34977e71-45e9-48e8-a11f-357705938038-proxy-tls\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.557205 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.557189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34977e71-45e9-48e8-a11f-357705938038-kserve-provision-location\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.557397 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.557282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4skz\" (UniqueName: \"kubernetes.io/projected/34977e71-45e9-48e8-a11f-357705938038-kube-api-access-w4skz\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.557397 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.557364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-cabundle-cert\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.658452 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.658418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-cabundle-cert\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.658642 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.658457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.658642 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.658482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34977e71-45e9-48e8-a11f-357705938038-proxy-tls\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.658642 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.658513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34977e71-45e9-48e8-a11f-357705938038-kserve-provision-location\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.658642 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.658561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4skz\" (UniqueName: \"kubernetes.io/projected/34977e71-45e9-48e8-a11f-357705938038-kube-api-access-w4skz\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.658995 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.658970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34977e71-45e9-48e8-a11f-357705938038-kserve-provision-location\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.659171 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.659155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-cabundle-cert\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.659225 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.659206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.661183 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.661161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34977e71-45e9-48e8-a11f-357705938038-proxy-tls\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.668298 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.668278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4skz\" (UniqueName: \"kubernetes.io/projected/34977e71-45e9-48e8-a11f-357705938038-kube-api-access-w4skz\") pod \"raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") " pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:09.814872 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:09.814791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:10.088975 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:10.088949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"]
Sep 16 18:58:10.091082 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:58:10.091053    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34977e71_45e9_48e8_a11f_357705938038.slice/crio-2d3962600ac72db0d81edb5d2a693545bb58c2e6cd09bae73fa481b53e2b059f WatchSource:0}: Error finding container 2d3962600ac72db0d81edb5d2a693545bb58c2e6cd09bae73fa481b53e2b059f: Status 404 returned error can't find the container with id 2d3962600ac72db0d81edb5d2a693545bb58c2e6cd09bae73fa481b53e2b059f
Sep 16 18:58:10.709582 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:10.709544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" event={"ID":"34977e71-45e9-48e8-a11f-357705938038","Type":"ContainerStarted","Data":"8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f"}
Sep 16 18:58:10.710078 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:10.709601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" event={"ID":"34977e71-45e9-48e8-a11f-357705938038","Type":"ContainerStarted","Data":"2d3962600ac72db0d81edb5d2a693545bb58c2e6cd09bae73fa481b53e2b059f"}
Sep 16 18:58:14.281437 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:14.281391    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:14.281893 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:14.281547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:58:14.727061 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:14.727025    2578 generic.go:356] "Generic (PLEG): container finished" podID="34977e71-45e9-48e8-a11f-357705938038" containerID="8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f" exitCode=0
Sep 16 18:58:14.727232 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:14.727086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" event={"ID":"34977e71-45e9-48e8-a11f-357705938038","Type":"ContainerDied","Data":"8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f"}
Sep 16 18:58:15.733533 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:15.733498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" event={"ID":"34977e71-45e9-48e8-a11f-357705938038","Type":"ContainerStarted","Data":"6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1"}
Sep 16 18:58:15.733876 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:15.733561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" event={"ID":"34977e71-45e9-48e8-a11f-357705938038","Type":"ContainerStarted","Data":"d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01"}
Sep 16 18:58:15.733876 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:15.733842    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:15.734044 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:15.734028    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:15.735450 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:15.735421    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:58:15.757749 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:15.757706    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podStartSLOduration=2.711272044 podStartE2EDuration="6.757691147s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.046419103s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:15.756049498 +0000 UTC m=+1048.886596210" watchObservedRunningTime="2026-09-16 18:58:15.757691147 +0000 UTC m=+1048.888237858"
Sep 16 18:58:16.738416 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:16.738374    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:58:19.280783 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:19.280738    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:19.285839 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:19.285806    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:19.286963 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:19.286940    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:58:21.742908 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:21.742874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:58:21.743517 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:21.743488    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:58:24.281107 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:24.281063    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:29.280765 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:29.280718    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:29.285768 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:29.285737    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:29.285901 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:29.285866    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:58:29.287944 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:29.287923    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:58:29.288055 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:29.288040    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:58:31.307207 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.307177    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:58:31.441208 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.441116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-cabundle-cert\") pod \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") "
Sep 16 18:58:31.441208 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.441159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kserve-provision-location\") pod \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") "
Sep 16 18:58:31.441438 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.441227    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5h77\" (UniqueName: \"kubernetes.io/projected/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-api-access-v5h77\") pod \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") "
Sep 16 18:58:31.441438 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.441258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-rbac-proxy-sar-config\") pod \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") "
Sep 16 18:58:31.441438 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.441295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-proxy-tls\") pod \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\" (UID: \"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15\") "
Sep 16 18:58:31.441563 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.441536    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" (UID: "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:31.441563 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.441541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" (UID: "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:31.441638 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.441606    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" (UID: "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:31.443630 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.443598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" (UID: "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:31.443715 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.443606    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-api-access-v5h77" (OuterVolumeSpecName: "kube-api-access-v5h77") pod "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" (UID: "fa10fe58-364e-4036-a5f3-ce0b7bfb6a15"). InnerVolumeSpecName "kube-api-access-v5h77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:31.542742 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.542702    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:58:31.542742 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.542734    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:58:31.542742 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.542744    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5h77\" (UniqueName: \"kubernetes.io/projected/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-api-access-v5h77\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:58:31.542742 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.542754    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:58:31.543034 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.542763    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:58:31.744296 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.744256    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:58:31.804468 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.804436    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerID="b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a" exitCode=0
Sep 16 18:58:31.804620 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.804495    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"
Sep 16 18:58:31.804620 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.804522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerDied","Data":"b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a"}
Sep 16 18:58:31.804620 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.804600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf" event={"ID":"fa10fe58-364e-4036-a5f3-ce0b7bfb6a15","Type":"ContainerDied","Data":"c2680138bf3d8c155dd5583b51bcfe5479a6ae31f7ec3b1b774ba4c8337ff123"}
Sep 16 18:58:31.804741 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.804639    2578 scope.go:117] "RemoveContainer" containerID="b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a"
Sep 16 18:58:31.814766 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.814744    2578 scope.go:117] "RemoveContainer" containerID="2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0"
Sep 16 18:58:31.823209 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.823190    2578 scope.go:117] "RemoveContainer" containerID="8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537"
Sep 16 18:58:31.829298 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.829271    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"]
Sep 16 18:58:31.830848 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.830828    2578 scope.go:117] "RemoveContainer" containerID="76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f"
Sep 16 18:58:31.834146 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.834121    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f5e4d-predictor-84c785b866-n8zvf"]
Sep 16 18:58:31.838663 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.838644    2578 scope.go:117] "RemoveContainer" containerID="b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a"
Sep 16 18:58:31.838967 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:58:31.838949    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a\": container with ID starting with b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a not found: ID does not exist" containerID="b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a"
Sep 16 18:58:31.839046 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.838975    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a"} err="failed to get container status \"b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a\": rpc error: code = NotFound desc = could not find container \"b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a\": container with ID starting with b2a3cfe5ca1600ab3f3db61bd449959100c5a31fa6af1028f147b0575b35043a not found: ID does not exist"
Sep 16 18:58:31.839046 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.838993    2578 scope.go:117] "RemoveContainer" containerID="2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0"
Sep 16 18:58:31.839297 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:58:31.839279    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0\": container with ID starting with 2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0 not found: ID does not exist" containerID="2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0"
Sep 16 18:58:31.839348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.839305    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0"} err="failed to get container status \"2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0\": rpc error: code = NotFound desc = could not find container \"2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0\": container with ID starting with 2aeefae1308e274094c9ec8f8d110ffd9a5aa540d8eec4f004d5b8e3c2fed3b0 not found: ID does not exist"
Sep 16 18:58:31.839348 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.839321    2578 scope.go:117] "RemoveContainer" containerID="8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537"
Sep 16 18:58:31.839567 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:58:31.839548    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537\": container with ID starting with 8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537 not found: ID does not exist" containerID="8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537"
Sep 16 18:58:31.839609 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.839573    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537"} err="failed to get container status \"8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537\": rpc error: code = NotFound desc = could not find container \"8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537\": container with ID starting with 8c63ffba4c51c36701e34023270ac6e5d27f3aaf50d92cec7c288ad192067537 not found: ID does not exist"
Sep 16 18:58:31.839609 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.839589    2578 scope.go:117] "RemoveContainer" containerID="76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f"
Sep 16 18:58:31.839840 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:58:31.839823    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f\": container with ID starting with 76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f not found: ID does not exist" containerID="76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f"
Sep 16 18:58:31.839880 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:31.839845    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f"} err="failed to get container status \"76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f\": rpc error: code = NotFound desc = could not find container \"76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f\": container with ID starting with 76e12f3eb761f0a0d6ec90189811a0edd2c4ad847fe321d0fc07070cd406858f not found: ID does not exist"
Sep 16 18:58:33.331890 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:33.331854    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" path="/var/lib/kubelet/pods/fa10fe58-364e-4036-a5f3-ce0b7bfb6a15/volumes"
Sep 16 18:58:41.743403 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:41.743360    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:58:51.744237 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:58:51.744189    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:59:01.743692 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:01.743645    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:59:11.743507 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:11.743468    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:59:21.743594 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:21.743552    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:59:23.327730 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:23.327688    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:59:33.331111 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:33.331082    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:59:39.699118 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:39.699083    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"]
Sep 16 18:59:39.699508 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:39.699331    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" containerID="cri-o://d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01" gracePeriod=30
Sep 16 18:59:39.699508 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:39.699360    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kube-rbac-proxy" containerID="cri-o://6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1" gracePeriod=30
Sep 16 18:59:40.073820 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:40.073784    2578 generic.go:356] "Generic (PLEG): container finished" podID="34977e71-45e9-48e8-a11f-357705938038" containerID="6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1" exitCode=2
Sep 16 18:59:40.073996 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:40.073855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" event={"ID":"34977e71-45e9-48e8-a11f-357705938038","Type":"ContainerDied","Data":"6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1"}
Sep 16 18:59:41.738900 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:41.738854    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 16 18:59:43.328282 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:43.328240    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 16 18:59:44.447338 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.447312    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:59:44.560388 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.560298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-cabundle-cert\") pod \"34977e71-45e9-48e8-a11f-357705938038\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") "
Sep 16 18:59:44.560388 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.560349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-kube-rbac-proxy-sar-config\") pod \"34977e71-45e9-48e8-a11f-357705938038\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") "
Sep 16 18:59:44.560612 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.560401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4skz\" (UniqueName: \"kubernetes.io/projected/34977e71-45e9-48e8-a11f-357705938038-kube-api-access-w4skz\") pod \"34977e71-45e9-48e8-a11f-357705938038\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") "
Sep 16 18:59:44.560612 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.560445    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34977e71-45e9-48e8-a11f-357705938038-proxy-tls\") pod \"34977e71-45e9-48e8-a11f-357705938038\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") "
Sep 16 18:59:44.560612 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.560472    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34977e71-45e9-48e8-a11f-357705938038-kserve-provision-location\") pod \"34977e71-45e9-48e8-a11f-357705938038\" (UID: \"34977e71-45e9-48e8-a11f-357705938038\") "
Sep 16 18:59:44.560836 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.560814    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34977e71-45e9-48e8-a11f-357705938038" (UID: "34977e71-45e9-48e8-a11f-357705938038"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:44.560889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.560819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34977e71-45e9-48e8-a11f-357705938038-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34977e71-45e9-48e8-a11f-357705938038" (UID: "34977e71-45e9-48e8-a11f-357705938038"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:44.560889 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.560837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34977e71-45e9-48e8-a11f-357705938038" (UID: "34977e71-45e9-48e8-a11f-357705938038"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:44.562785 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.562751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34977e71-45e9-48e8-a11f-357705938038-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34977e71-45e9-48e8-a11f-357705938038" (UID: "34977e71-45e9-48e8-a11f-357705938038"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:44.562892 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.562779    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34977e71-45e9-48e8-a11f-357705938038-kube-api-access-w4skz" (OuterVolumeSpecName: "kube-api-access-w4skz") pod "34977e71-45e9-48e8-a11f-357705938038" (UID: "34977e71-45e9-48e8-a11f-357705938038"). InnerVolumeSpecName "kube-api-access-w4skz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:44.661704 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.661666    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34977e71-45e9-48e8-a11f-357705938038-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:59:44.661704 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.661697    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34977e71-45e9-48e8-a11f-357705938038-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:59:44.661704 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.661711    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:59:44.661931 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.661724    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34977e71-45e9-48e8-a11f-357705938038-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:59:44.661931 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:44.661738    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4skz\" (UniqueName: \"kubernetes.io/projected/34977e71-45e9-48e8-a11f-357705938038-kube-api-access-w4skz\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 18:59:45.001096 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001053    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"]
Sep 16 18:59:45.001423 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001405    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent"
Sep 16 18:59:45.001506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001426    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent"
Sep 16 18:59:45.001506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001448    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container"
Sep 16 18:59:45.001506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001457    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container"
Sep 16 18:59:45.001506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001467    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy"
Sep 16 18:59:45.001506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001477    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy"
Sep 16 18:59:45.001506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001486    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kube-rbac-proxy"
Sep 16 18:59:45.001506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001494    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kube-rbac-proxy"
Sep 16 18:59:45.001506 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001505    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001513    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001523    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="storage-initializer"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001532    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="storage-initializer"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001543    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="storage-initializer"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001551    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="storage-initializer"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001640    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kserve-container"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001653    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="agent"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001663    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="34977e71-45e9-48e8-a11f-357705938038" containerName="kube-rbac-proxy"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001673    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kserve-container"
Sep 16 18:59:45.001894 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.001684    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa10fe58-364e-4036-a5f3-ce0b7bfb6a15" containerName="kube-rbac-proxy"
Sep 16 18:59:45.004958 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.004936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.010050 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.010027    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-da400-predictor-serving-cert\""
Sep 16 18:59:45.011380 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.011346    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-da400-kube-rbac-proxy-sar-config\""
Sep 16 18:59:45.033705 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.033682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"]
Sep 16 18:59:45.064491 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.064453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kserve-provision-location\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.064658 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.064511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.064658 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.064571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ss6j\" (UniqueName: \"kubernetes.io/projected/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-api-access-9ss6j\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.064762 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.064657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.064762 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.064685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-cabundle-cert\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.093715 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.093678    2578 generic.go:356] "Generic (PLEG): container finished" podID="34977e71-45e9-48e8-a11f-357705938038" containerID="d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01" exitCode=0
Sep 16 18:59:45.093843 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.093742    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"
Sep 16 18:59:45.093843 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.093766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" event={"ID":"34977e71-45e9-48e8-a11f-357705938038","Type":"ContainerDied","Data":"d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01"}
Sep 16 18:59:45.093843 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.093829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf" event={"ID":"34977e71-45e9-48e8-a11f-357705938038","Type":"ContainerDied","Data":"2d3962600ac72db0d81edb5d2a693545bb58c2e6cd09bae73fa481b53e2b059f"}
Sep 16 18:59:45.093939 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.093858    2578 scope.go:117] "RemoveContainer" containerID="6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1"
Sep 16 18:59:45.102207 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.102188    2578 scope.go:117] "RemoveContainer" containerID="d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01"
Sep 16 18:59:45.109840 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.109821    2578 scope.go:117] "RemoveContainer" containerID="8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f"
Sep 16 18:59:45.117443 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.117425    2578 scope.go:117] "RemoveContainer" containerID="6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1"
Sep 16 18:59:45.117730 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:59:45.117711    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1\": container with ID starting with 6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1 not found: ID does not exist" containerID="6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1"
Sep 16 18:59:45.117779 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.117739    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1"} err="failed to get container status \"6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1\": rpc error: code = NotFound desc = could not find container \"6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1\": container with ID starting with 6862cc813b02816d2f4e41293f1eb7efb0dff3eaee9d93664b4ebf5f90b1d6b1 not found: ID does not exist"
Sep 16 18:59:45.117779 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.117756    2578 scope.go:117] "RemoveContainer" containerID="d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01"
Sep 16 18:59:45.118042 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:59:45.118020    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01\": container with ID starting with d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01 not found: ID does not exist" containerID="d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01"
Sep 16 18:59:45.118097 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.118055    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01"} err="failed to get container status \"d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01\": rpc error: code = NotFound desc = could not find container \"d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01\": container with ID starting with d049a4512e88356e75401132b20969115218f666e484f5150d02bdef8f7dbc01 not found: ID does not exist"
Sep 16 18:59:45.118097 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.118073    2578 scope.go:117] "RemoveContainer" containerID="8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f"
Sep 16 18:59:45.118328 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:59:45.118311    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f\": container with ID starting with 8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f not found: ID does not exist" containerID="8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f"
Sep 16 18:59:45.118381 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.118337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f"} err="failed to get container status \"8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f\": rpc error: code = NotFound desc = could not find container \"8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f\": container with ID starting with 8fc0aa68d337f5f37b86a44fb9c487853e345b1f4da230fd26b607671757c41f not found: ID does not exist"
Sep 16 18:59:45.127639 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.127610    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"]
Sep 16 18:59:45.137283 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.137259    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fded7-predictor-7f6c8b557f-9jrlf"]
Sep 16 18:59:45.166041 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.165985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.166212 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.166059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ss6j\" (UniqueName: \"kubernetes.io/projected/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-api-access-9ss6j\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.166212 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.166102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.166212 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.166119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-cabundle-cert\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.166212 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:59:45.166151    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-serving-cert: secret "raw-sklearn-runtime-da400-predictor-serving-cert" not found
Sep 16 18:59:45.166433 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.166209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kserve-provision-location\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.166433 ip-10-0-143-155 kubenswrapper[2578]: E0916 18:59:45.166296    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls podName:0796b9f1-c214-4ffe-8d8c-7f3c849a0efe nodeName:}" failed. No retries permitted until 2026-09-16 18:59:45.666274773 +0000 UTC m=+1138.796821462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls") pod "raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" (UID: "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe") : secret "raw-sklearn-runtime-da400-predictor-serving-cert" not found
Sep 16 18:59:45.166651 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.166631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kserve-provision-location\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.166823 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.166806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-cabundle-cert\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.166863 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.166844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.192637 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.192605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ss6j\" (UniqueName: \"kubernetes.io/projected/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-api-access-9ss6j\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.331681 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.331599    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34977e71-45e9-48e8-a11f-357705938038" path="/var/lib/kubelet/pods/34977e71-45e9-48e8-a11f-357705938038/volumes"
Sep 16 18:59:45.670951 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.670859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.673482 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.673462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls\") pod \"raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:45.915172 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:45.915137    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:46.203931 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:46.203837    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"]
Sep 16 18:59:46.208543 ip-10-0-143-155 kubenswrapper[2578]: W0916 18:59:46.208512    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0796b9f1_c214_4ffe_8d8c_7f3c849a0efe.slice/crio-92c19b9923bbb7c0021f136bf3bd16ba5cf64bfa67bab3aa17c5c6096e6e36c8 WatchSource:0}: Error finding container 92c19b9923bbb7c0021f136bf3bd16ba5cf64bfa67bab3aa17c5c6096e6e36c8: Status 404 returned error can't find the container with id 92c19b9923bbb7c0021f136bf3bd16ba5cf64bfa67bab3aa17c5c6096e6e36c8
Sep 16 18:59:47.109633 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:47.109596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" event={"ID":"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe","Type":"ContainerStarted","Data":"3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c"}
Sep 16 18:59:47.110133 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:47.109663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" event={"ID":"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe","Type":"ContainerStarted","Data":"92c19b9923bbb7c0021f136bf3bd16ba5cf64bfa67bab3aa17c5c6096e6e36c8"}
Sep 16 18:59:50.127462 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:50.127422    2578 generic.go:356] "Generic (PLEG): container finished" podID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerID="3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c" exitCode=0
Sep 16 18:59:50.127908 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:50.127500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" event={"ID":"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe","Type":"ContainerDied","Data":"3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c"}
Sep 16 18:59:51.133940 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:51.133898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" event={"ID":"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe","Type":"ContainerStarted","Data":"e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98"}
Sep 16 18:59:51.134323 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:51.133971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" event={"ID":"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe","Type":"ContainerStarted","Data":"da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d"}
Sep 16 18:59:51.134323 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:51.134181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:51.161631 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:51.161580    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podStartSLOduration=3.452713953 podStartE2EDuration="7.161564442s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.708850489s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:51.159252867 +0000 UTC m=+1144.289799579" watchObservedRunningTime="2026-09-16 18:59:51.161564442 +0000 UTC m=+1144.292111153"
Sep 16 18:59:52.137961 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:52.137933    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:52.139450 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:52.139421    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:59:53.142076 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:53.142042    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:59:58.146648 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:58.146620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 18:59:58.147257 ip-10-0-143-155 kubenswrapper[2578]: I0916 18:59:58.147224    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:08.147408 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:08.147366    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:18.147621 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:18.147532    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:28.147191 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:28.147152    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:38.147717 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:38.147671    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:46.649390 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.649347    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"]
Sep 16 19:00:46.652659 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.652643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.655281 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.655253    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:00:46.655395 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.655304    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:00:46.673833 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.673813    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"]
Sep 16 19:00:46.737868 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.737835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk89z\" (UniqueName: \"kubernetes.io/projected/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-api-access-zk89z\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.738059 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.737887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.738059 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.737918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.738059 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.737941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.738182 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.738077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f3751a-3c49-4069-9450-b124ab6343ef-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.792220 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.792189    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"]
Sep 16 19:00:46.795550 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.795534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.798295 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.798270    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:00:46.815631 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.815606    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"]
Sep 16 19:00:46.838845 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.838817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee39f02e-c0af-4752-bd1c-4407fec155f1-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.838845 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.838853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee39f02e-c0af-4752-bd1c-4407fec155f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.839065 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.838881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f3751a-3c49-4069-9450-b124ab6343ef-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.839065 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.838932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.839065 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.839065 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5k8\" (UniqueName: \"kubernetes.io/projected/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-api-access-dq5k8\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.839234 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk89z\" (UniqueName: \"kubernetes.io/projected/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-api-access-zk89z\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.839234 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.839234 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.839234 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.839234 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f3751a-3c49-4069-9450-b124ab6343ef-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.839481 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:00:46.839306    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:00:46.839481 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:00:46.839388    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls podName:a9f3751a-3c49-4069-9450-b124ab6343ef nodeName:}" failed. No retries permitted until 2026-09-16 19:00:47.339367196 +0000 UTC m=+1200.469913898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls") pod "isvc-canary-create-v2-predictor-85b98c9747-gwkps" (UID: "a9f3751a-3c49-4069-9450-b124ab6343ef") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:00:46.839779 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.839827 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.839786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.853360 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.853334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk89z\" (UniqueName: \"kubernetes.io/projected/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-api-access-zk89z\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:46.940033 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.939935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee39f02e-c0af-4752-bd1c-4407fec155f1-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.940033 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.939970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee39f02e-c0af-4752-bd1c-4407fec155f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.940033 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.940000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.940241 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.940056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.940241 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.940078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5k8\" (UniqueName: \"kubernetes.io/projected/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-api-access-dq5k8\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.940392 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.940371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee39f02e-c0af-4752-bd1c-4407fec155f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.940687 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.940668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.940764 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.940745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.942687 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.942671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee39f02e-c0af-4752-bd1c-4407fec155f1-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:46.950237 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:46.950215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5k8\" (UniqueName: \"kubernetes.io/projected/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-api-access-dq5k8\") pod \"isvc-canary-create-predictor-76cc455874-cdvhn\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:47.105790 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:47.105753    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:47.298502 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:47.298477    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 19:00:47.301892 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:47.301867    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 19:00:47.342962 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:47.342923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:47.345647 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:47.345617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-gwkps\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:47.385252 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:47.385230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"]
Sep 16 19:00:47.387886 ip-10-0-143-155 kubenswrapper[2578]: W0916 19:00:47.387864    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee39f02e_c0af_4752_bd1c_4407fec155f1.slice/crio-206cf995f37f4824a84a62c03efc3a6fda359bebb0832d057f1aa758b9ccc5d9 WatchSource:0}: Error finding container 206cf995f37f4824a84a62c03efc3a6fda359bebb0832d057f1aa758b9ccc5d9: Status 404 returned error can't find the container with id 206cf995f37f4824a84a62c03efc3a6fda359bebb0832d057f1aa758b9ccc5d9
Sep 16 19:00:47.562902 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:47.562814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:47.841672 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:47.841647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"]
Sep 16 19:00:47.844735 ip-10-0-143-155 kubenswrapper[2578]: W0916 19:00:47.844701    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f3751a_3c49_4069_9450_b124ab6343ef.slice/crio-e3d992991e177d64d5e71c2a0864f98e8ba47b54bd46061c7e333492d2e4b635 WatchSource:0}: Error finding container e3d992991e177d64d5e71c2a0864f98e8ba47b54bd46061c7e333492d2e4b635: Status 404 returned error can't find the container with id e3d992991e177d64d5e71c2a0864f98e8ba47b54bd46061c7e333492d2e4b635
Sep 16 19:00:48.147844 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:48.147759    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:48.355732 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:48.355689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" event={"ID":"a9f3751a-3c49-4069-9450-b124ab6343ef","Type":"ContainerStarted","Data":"4164431e4985faac0a108aa1b834a5fd31cc52354438051e99f9c765a856b960"}
Sep 16 19:00:48.355931 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:48.355816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" event={"ID":"a9f3751a-3c49-4069-9450-b124ab6343ef","Type":"ContainerStarted","Data":"e3d992991e177d64d5e71c2a0864f98e8ba47b54bd46061c7e333492d2e4b635"}
Sep 16 19:00:48.357378 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:48.357348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" event={"ID":"ee39f02e-c0af-4752-bd1c-4407fec155f1","Type":"ContainerStarted","Data":"adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532"}
Sep 16 19:00:48.357487 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:48.357418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" event={"ID":"ee39f02e-c0af-4752-bd1c-4407fec155f1","Type":"ContainerStarted","Data":"206cf995f37f4824a84a62c03efc3a6fda359bebb0832d057f1aa758b9ccc5d9"}
Sep 16 19:00:51.372084 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:51.371978    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerID="adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532" exitCode=0
Sep 16 19:00:51.372084 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:51.372035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" event={"ID":"ee39f02e-c0af-4752-bd1c-4407fec155f1","Type":"ContainerDied","Data":"adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532"}
Sep 16 19:00:52.378236 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:52.378205    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerID="4164431e4985faac0a108aa1b834a5fd31cc52354438051e99f9c765a856b960" exitCode=0
Sep 16 19:00:52.378782 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:52.378282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" event={"ID":"a9f3751a-3c49-4069-9450-b124ab6343ef","Type":"ContainerDied","Data":"4164431e4985faac0a108aa1b834a5fd31cc52354438051e99f9c765a856b960"}
Sep 16 19:00:52.380792 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:52.380766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" event={"ID":"ee39f02e-c0af-4752-bd1c-4407fec155f1","Type":"ContainerStarted","Data":"b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff"}
Sep 16 19:00:52.380906 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:52.380885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" event={"ID":"ee39f02e-c0af-4752-bd1c-4407fec155f1","Type":"ContainerStarted","Data":"408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30"}
Sep 16 19:00:52.381111 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:52.381092    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:52.381174 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:52.381154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:52.382583 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:52.382558    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:00:52.439628 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:52.439348    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podStartSLOduration=2.771505307 podStartE2EDuration="6.439325431s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.667820124s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:52.438919746 +0000 UTC m=+1205.569466458" watchObservedRunningTime="2026-09-16 19:00:52.439325431 +0000 UTC m=+1205.569872142"
Sep 16 19:00:53.386726 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:53.386688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" event={"ID":"a9f3751a-3c49-4069-9450-b124ab6343ef","Type":"ContainerStarted","Data":"a3bc3539dc8d2b3e1fbcbce7fbc2e092dee00e05fc4b4c2cadc5c5bc81e35229"}
Sep 16 19:00:53.387141 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:53.386755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" event={"ID":"a9f3751a-3c49-4069-9450-b124ab6343ef","Type":"ContainerStarted","Data":"e8399b9452ed243f0bc83d1d2904835ef13d48e2b2a1b350f5d5c42ba1a3addc"}
Sep 16 19:00:53.387141 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:53.387034    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:00:53.387141 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:53.387087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:53.421708 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:53.421657    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podStartSLOduration=3.446263292 podStartE2EDuration="7.421641063s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975377771s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:53.418821991 +0000 UTC m=+1206.549368704" watchObservedRunningTime="2026-09-16 19:00:53.421641063 +0000 UTC m=+1206.552187773"
Sep 16 19:00:54.392414 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:54.392381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:00:54.393777 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:54.393748    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:00:55.397285 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:55.397250    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:00:58.147338 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:58.147299    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:58.392229 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:58.392199    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:00:58.392689 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:00:58.392665    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:01:00.402259 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:00.402230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:01:00.402891 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:00.402860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:01.327603 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:01.327559    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:08.393031 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:08.392968    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:01:10.402851 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:10.402810    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:11.332165 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:11.332136    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 19:01:14.904684 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:14.904652    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"]
Sep 16 19:01:14.905204 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:14.904939    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container" containerID="cri-o://da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d" gracePeriod=30
Sep 16 19:01:14.905204 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:14.905044    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kube-rbac-proxy" containerID="cri-o://e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98" gracePeriod=30
Sep 16 19:01:15.484273 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:15.484236    2578 generic.go:356] "Generic (PLEG): container finished" podID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerID="e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98" exitCode=2
Sep 16 19:01:15.484448 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:15.484311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" event={"ID":"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe","Type":"ContainerDied","Data":"e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98"}
Sep 16 19:01:18.142653 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:18.142609    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 19:01:18.393565 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:18.393478    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:01:19.860400 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.860377    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 19:01:19.982258 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.982229    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-cabundle-cert\") pod \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") "
Sep 16 19:01:19.982428 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.982291    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ss6j\" (UniqueName: \"kubernetes.io/projected/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-api-access-9ss6j\") pod \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") "
Sep 16 19:01:19.982428 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.982334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kserve-provision-location\") pod \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") "
Sep 16 19:01:19.982428 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.982370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls\") pod \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") "
Sep 16 19:01:19.982428 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.982405    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-rbac-proxy-sar-config\") pod \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\" (UID: \"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe\") "
Sep 16 19:01:19.982745 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.982711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" (UID: "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:19.982745 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.982662    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" (UID: "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:19.982864 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.982807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" (UID: "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:19.984672 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.984653    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" (UID: "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:19.984731 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:19.984708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-api-access-9ss6j" (OuterVolumeSpecName: "kube-api-access-9ss6j") pod "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" (UID: "0796b9f1-c214-4ffe-8d8c-7f3c849a0efe"). InnerVolumeSpecName "kube-api-access-9ss6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:20.083551 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.083506    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.083551 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.083540    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.083551 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.083549    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.083551 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.083558    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ss6j\" (UniqueName: \"kubernetes.io/projected/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kube-api-access-9ss6j\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.083830 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.083568    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.403173 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.403083    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:20.506467 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.506432    2578 generic.go:356] "Generic (PLEG): container finished" podID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerID="da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d" exitCode=0
Sep 16 19:01:20.506639 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.506491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" event={"ID":"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe","Type":"ContainerDied","Data":"da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d"}
Sep 16 19:01:20.506639 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.506507    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"
Sep 16 19:01:20.506639 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.506570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c" event={"ID":"0796b9f1-c214-4ffe-8d8c-7f3c849a0efe","Type":"ContainerDied","Data":"92c19b9923bbb7c0021f136bf3bd16ba5cf64bfa67bab3aa17c5c6096e6e36c8"}
Sep 16 19:01:20.506639 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.506597    2578 scope.go:117] "RemoveContainer" containerID="e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98"
Sep 16 19:01:20.515189 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.515166    2578 scope.go:117] "RemoveContainer" containerID="da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d"
Sep 16 19:01:20.525284 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.525259    2578 scope.go:117] "RemoveContainer" containerID="3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c"
Sep 16 19:01:20.533177 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.533152    2578 scope.go:117] "RemoveContainer" containerID="e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98"
Sep 16 19:01:20.533462 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:01:20.533441    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98\": container with ID starting with e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98 not found: ID does not exist" containerID="e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98"
Sep 16 19:01:20.533528 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.533471    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98"} err="failed to get container status \"e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98\": rpc error: code = NotFound desc = could not find container \"e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98\": container with ID starting with e694fe070a7b6e6521a4301ffc7040c0459b47b929f39a41f9c45bfbb5cf0c98 not found: ID does not exist"
Sep 16 19:01:20.533528 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.533489    2578 scope.go:117] "RemoveContainer" containerID="da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d"
Sep 16 19:01:20.533756 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:01:20.533739    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d\": container with ID starting with da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d not found: ID does not exist" containerID="da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d"
Sep 16 19:01:20.533799 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.533763    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d"} err="failed to get container status \"da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d\": rpc error: code = NotFound desc = could not find container \"da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d\": container with ID starting with da420a88c08db6f1d574923308558f5539910b97808f0969d27fc311def41f6d not found: ID does not exist"
Sep 16 19:01:20.533799 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.533778    2578 scope.go:117] "RemoveContainer" containerID="3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c"
Sep 16 19:01:20.533999 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:01:20.533982    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c\": container with ID starting with 3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c not found: ID does not exist" containerID="3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c"
Sep 16 19:01:20.534342 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.534022    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c"} err="failed to get container status \"3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c\": rpc error: code = NotFound desc = could not find container \"3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c\": container with ID starting with 3371bb698a5d4e17ce7925f03aaeb8c8f9878fc1c61a75cdbf9ed3ec9127a11c not found: ID does not exist"
Sep 16 19:01:20.538038 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.537997    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"]
Sep 16 19:01:20.545023 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:20.544974    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-da400-predictor-647cdd7f4c-h792c"]
Sep 16 19:01:21.332046 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:21.331997    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" path="/var/lib/kubelet/pods/0796b9f1-c214-4ffe-8d8c-7f3c849a0efe/volumes"
Sep 16 19:01:28.393208 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:28.393164    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:01:30.403058 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:30.402988    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:38.392636 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:38.392595    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:01:40.403543 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:40.403501    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:48.392827 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:48.392790    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:01:50.403549 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:50.403512    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:01:58.393343 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:01:58.393306    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:02:00.402872 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:00.402824    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:08.394172 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:08.394137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:02:10.404448 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:10.404411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:02:10.535179 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:10.535141    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"]
Sep 16 19:02:10.535458 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:10.535426    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container" containerID="cri-o://408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30" gracePeriod=30
Sep 16 19:02:10.535573 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:10.535495    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kube-rbac-proxy" containerID="cri-o://b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff" gracePeriod=30
Sep 16 19:02:10.728204 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:10.728175    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerID="b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff" exitCode=2
Sep 16 19:02:10.728389 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:10.728255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" event={"ID":"ee39f02e-c0af-4752-bd1c-4407fec155f1","Type":"ContainerDied","Data":"b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff"}
Sep 16 19:02:13.387743 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:13.387704    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 19:02:15.384322 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.384286    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:02:15.409940 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.409911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee39f02e-c0af-4752-bd1c-4407fec155f1-kserve-provision-location\") pod \"ee39f02e-c0af-4752-bd1c-4407fec155f1\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") "
Sep 16 19:02:15.410140 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.409970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-cabundle-cert\") pod \"ee39f02e-c0af-4752-bd1c-4407fec155f1\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") "
Sep 16 19:02:15.410140 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.410034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee39f02e-c0af-4752-bd1c-4407fec155f1-proxy-tls\") pod \"ee39f02e-c0af-4752-bd1c-4407fec155f1\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") "
Sep 16 19:02:15.410140 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.410080    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-rbac-proxy-sar-config\") pod \"ee39f02e-c0af-4752-bd1c-4407fec155f1\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") "
Sep 16 19:02:15.410140 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.410104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq5k8\" (UniqueName: \"kubernetes.io/projected/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-api-access-dq5k8\") pod \"ee39f02e-c0af-4752-bd1c-4407fec155f1\" (UID: \"ee39f02e-c0af-4752-bd1c-4407fec155f1\") "
Sep 16 19:02:15.410374 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.410317    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39f02e-c0af-4752-bd1c-4407fec155f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee39f02e-c0af-4752-bd1c-4407fec155f1" (UID: "ee39f02e-c0af-4752-bd1c-4407fec155f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:15.410426 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.410389    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee39f02e-c0af-4752-bd1c-4407fec155f1" (UID: "ee39f02e-c0af-4752-bd1c-4407fec155f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:15.410618 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.410585    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee39f02e-c0af-4752-bd1c-4407fec155f1" (UID: "ee39f02e-c0af-4752-bd1c-4407fec155f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:15.412888 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.412856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-api-access-dq5k8" (OuterVolumeSpecName: "kube-api-access-dq5k8") pod "ee39f02e-c0af-4752-bd1c-4407fec155f1" (UID: "ee39f02e-c0af-4752-bd1c-4407fec155f1"). InnerVolumeSpecName "kube-api-access-dq5k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:15.413111 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.413085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee39f02e-c0af-4752-bd1c-4407fec155f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee39f02e-c0af-4752-bd1c-4407fec155f1" (UID: "ee39f02e-c0af-4752-bd1c-4407fec155f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:15.510670 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.510628    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.510670 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.510661    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee39f02e-c0af-4752-bd1c-4407fec155f1-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.510670 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.510671    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.510670 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.510681    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5k8\" (UniqueName: \"kubernetes.io/projected/ee39f02e-c0af-4752-bd1c-4407fec155f1-kube-api-access-dq5k8\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.510952 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.510692    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee39f02e-c0af-4752-bd1c-4407fec155f1-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.751032 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.750984    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerID="408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30" exitCode=0
Sep 16 19:02:15.751231 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.751077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" event={"ID":"ee39f02e-c0af-4752-bd1c-4407fec155f1","Type":"ContainerDied","Data":"408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30"}
Sep 16 19:02:15.751231 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.751089    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"
Sep 16 19:02:15.751231 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.751187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn" event={"ID":"ee39f02e-c0af-4752-bd1c-4407fec155f1","Type":"ContainerDied","Data":"206cf995f37f4824a84a62c03efc3a6fda359bebb0832d057f1aa758b9ccc5d9"}
Sep 16 19:02:15.751370 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.751250    2578 scope.go:117] "RemoveContainer" containerID="b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff"
Sep 16 19:02:15.760148 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.760129    2578 scope.go:117] "RemoveContainer" containerID="408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30"
Sep 16 19:02:15.767592 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.767574    2578 scope.go:117] "RemoveContainer" containerID="adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532"
Sep 16 19:02:15.775279 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.775258    2578 scope.go:117] "RemoveContainer" containerID="b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff"
Sep 16 19:02:15.775579 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:02:15.775558    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff\": container with ID starting with b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff not found: ID does not exist" containerID="b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff"
Sep 16 19:02:15.775628 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.775590    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff"} err="failed to get container status \"b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff\": rpc error: code = NotFound desc = could not find container \"b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff\": container with ID starting with b212254e2f13ed712b49801b6842fc9b3548e6a08f2daf85bb39212dea2dfaff not found: ID does not exist"
Sep 16 19:02:15.775628 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.775610    2578 scope.go:117] "RemoveContainer" containerID="408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30"
Sep 16 19:02:15.775845 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:02:15.775829    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30\": container with ID starting with 408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30 not found: ID does not exist" containerID="408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30"
Sep 16 19:02:15.775894 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.775852    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30"} err="failed to get container status \"408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30\": rpc error: code = NotFound desc = could not find container \"408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30\": container with ID starting with 408e22830e9991274cbb26217bf6b0b41d204fde9376709e4ddf53eb93a71b30 not found: ID does not exist"
Sep 16 19:02:15.775894 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.775867    2578 scope.go:117] "RemoveContainer" containerID="adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532"
Sep 16 19:02:15.776143 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:02:15.776118    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532\": container with ID starting with adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532 not found: ID does not exist" containerID="adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532"
Sep 16 19:02:15.776242 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.776148    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532"} err="failed to get container status \"adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532\": rpc error: code = NotFound desc = could not find container \"adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532\": container with ID starting with adb273c133f6541b8c396468c16732703e94aa36e623555b4aaf80b57713b532 not found: ID does not exist"
Sep 16 19:02:15.778796 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.778773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"]
Sep 16 19:02:15.784320 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:15.784294    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-cdvhn"]
Sep 16 19:02:16.760537 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:16.760508    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"]
Sep 16 19:02:16.760984 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:16.760786    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" containerID="cri-o://e8399b9452ed243f0bc83d1d2904835ef13d48e2b2a1b350f5d5c42ba1a3addc" gracePeriod=30
Sep 16 19:02:16.760984 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:16.760802    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kube-rbac-proxy" containerID="cri-o://a3bc3539dc8d2b3e1fbcbce7fbc2e092dee00e05fc4b4c2cadc5c5bc81e35229" gracePeriod=30
Sep 16 19:02:17.331332 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:17.331294    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" path="/var/lib/kubelet/pods/ee39f02e-c0af-4752-bd1c-4407fec155f1/volumes"
Sep 16 19:02:17.764585 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:17.764556    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerID="a3bc3539dc8d2b3e1fbcbce7fbc2e092dee00e05fc4b4c2cadc5c5bc81e35229" exitCode=2
Sep 16 19:02:17.764985 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:17.764631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" event={"ID":"a9f3751a-3c49-4069-9450-b124ab6343ef","Type":"ContainerDied","Data":"a3bc3539dc8d2b3e1fbcbce7fbc2e092dee00e05fc4b4c2cadc5c5bc81e35229"}
Sep 16 19:02:20.398298 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:20.398258    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 19:02:20.403624 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:20.403601    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:21.785966 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.785927    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerID="e8399b9452ed243f0bc83d1d2904835ef13d48e2b2a1b350f5d5c42ba1a3addc" exitCode=0
Sep 16 19:02:21.786349 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.785989    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" event={"ID":"a9f3751a-3c49-4069-9450-b124ab6343ef","Type":"ContainerDied","Data":"e8399b9452ed243f0bc83d1d2904835ef13d48e2b2a1b350f5d5c42ba1a3addc"}
Sep 16 19:02:21.813020 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.812979    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:02:21.859784 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.859753    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls\") pod \"a9f3751a-3c49-4069-9450-b124ab6343ef\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") "
Sep 16 19:02:21.859784 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.859789    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-cabundle-cert\") pod \"a9f3751a-3c49-4069-9450-b124ab6343ef\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") "
Sep 16 19:02:21.860093 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.859839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-rbac-proxy-sar-config\") pod \"a9f3751a-3c49-4069-9450-b124ab6343ef\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") "
Sep 16 19:02:21.860093 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.859869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk89z\" (UniqueName: \"kubernetes.io/projected/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-api-access-zk89z\") pod \"a9f3751a-3c49-4069-9450-b124ab6343ef\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") "
Sep 16 19:02:21.860093 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.859898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f3751a-3c49-4069-9450-b124ab6343ef-kserve-provision-location\") pod \"a9f3751a-3c49-4069-9450-b124ab6343ef\" (UID: \"a9f3751a-3c49-4069-9450-b124ab6343ef\") "
Sep 16 19:02:21.860352 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.860310    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f3751a-3c49-4069-9450-b124ab6343ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9f3751a-3c49-4069-9450-b124ab6343ef" (UID: "a9f3751a-3c49-4069-9450-b124ab6343ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:21.860352 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.860319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9f3751a-3c49-4069-9450-b124ab6343ef" (UID: "a9f3751a-3c49-4069-9450-b124ab6343ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:21.860352 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.860313    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9f3751a-3c49-4069-9450-b124ab6343ef" (UID: "a9f3751a-3c49-4069-9450-b124ab6343ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:21.862215 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.862182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9f3751a-3c49-4069-9450-b124ab6343ef" (UID: "a9f3751a-3c49-4069-9450-b124ab6343ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:21.862315 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.862246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-api-access-zk89z" (OuterVolumeSpecName: "kube-api-access-zk89z") pod "a9f3751a-3c49-4069-9450-b124ab6343ef" (UID: "a9f3751a-3c49-4069-9450-b124ab6343ef"). InnerVolumeSpecName "kube-api-access-zk89z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:21.961329 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.961242    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9f3751a-3c49-4069-9450-b124ab6343ef-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:21.961329 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.961272    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9f3751a-3c49-4069-9450-b124ab6343ef-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:21.961329 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.961282    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:21.961329 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.961290    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:21.961329 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.961298    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk89z\" (UniqueName: \"kubernetes.io/projected/a9f3751a-3c49-4069-9450-b124ab6343ef-kube-api-access-zk89z\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:02:21.978032 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.977990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"]
Sep 16 19:02:21.978287 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978274    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container"
Sep 16 19:02:21.978327 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978289    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container"
Sep 16 19:02:21.978327 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978298    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="storage-initializer"
Sep 16 19:02:21.978327 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978304    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="storage-initializer"
Sep 16 19:02:21.978327 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978311    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container"
Sep 16 19:02:21.978327 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978317    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container"
Sep 16 19:02:21.978327 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978327    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="storage-initializer"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978333    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="storage-initializer"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978339    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kube-rbac-proxy"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978345    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kube-rbac-proxy"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978352    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978357    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978363    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kube-rbac-proxy"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978369    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kube-rbac-proxy"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978379    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kube-rbac-proxy"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978385    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kube-rbac-proxy"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978393    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="storage-initializer"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978398    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="storage-initializer"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978444    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kserve-container"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978450    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" containerName="kube-rbac-proxy"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978456    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kserve-container"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978462    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee39f02e-c0af-4752-bd1c-4407fec155f1" containerName="kube-rbac-proxy"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978468    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kserve-container"
Sep 16 19:02:21.978504 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.978473    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0796b9f1-c214-4ffe-8d8c-7f3c849a0efe" containerName="kube-rbac-proxy"
Sep 16 19:02:21.981732 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.981710    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"]
Sep 16 19:02:21.981851 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.981832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:21.984682 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.984663    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:02:21.984778 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.984711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:21.985090 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.985074    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:02:21.991217 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:21.991195    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:02:22.002520 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.002497    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"]
Sep 16 19:02:22.021560 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.021525    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"]
Sep 16 19:02:22.062399 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abf83e4-61a6-45bb-8428-a47782b8c281-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.062608 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjmk\" (UniqueName: \"kubernetes.io/projected/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-api-access-bjjmk\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.062608 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.062608 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.062608 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.062831 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abf83e4-61a6-45bb-8428-a47782b8c281-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.062831 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8hv\" (UniqueName: \"kubernetes.io/projected/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-api-access-gf8hv\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.062831 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.062831 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.062831 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.062764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.163617 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.163617 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.163904 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abf83e4-61a6-45bb-8428-a47782b8c281-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.163904 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8hv\" (UniqueName: \"kubernetes.io/projected/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-api-access-gf8hv\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.163904 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.163904 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.163904 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.163904 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abf83e4-61a6-45bb-8428-a47782b8c281-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.163904 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjmk\" (UniqueName: \"kubernetes.io/projected/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-api-access-bjjmk\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.163904 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.163877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.164342 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.164295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.164584 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.164556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.164773 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.164750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abf83e4-61a6-45bb-8428-a47782b8c281-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.164896 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.164871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.164960 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.164912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.165057 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.164998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.166631 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.166612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.166963 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.166947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abf83e4-61a6-45bb-8428-a47782b8c281-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.174442 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.174419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8hv\" (UniqueName: \"kubernetes.io/projected/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-api-access-gf8hv\") pod \"isvc-canary-promote-predictor-6b6f66ff94-lmkzv\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.174549 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.174460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjmk\" (UniqueName: \"kubernetes.io/projected/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-api-access-bjjmk\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.293348 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.293313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:22.298907 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.298883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:22.575916 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.575834    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"]
Sep 16 19:02:22.579056 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.578981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"]
Sep 16 19:02:22.580129 ip-10-0-143-155 kubenswrapper[2578]: W0916 19:02:22.580107    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abf83e4_61a6_45bb_8428_a47782b8c281.slice/crio-432abb7bc82bb31fe737f9079970bd76fbc692a72668c5c96e70fa1eb3ad9dd7 WatchSource:0}: Error finding container 432abb7bc82bb31fe737f9079970bd76fbc692a72668c5c96e70fa1eb3ad9dd7: Status 404 returned error can't find the container with id 432abb7bc82bb31fe737f9079970bd76fbc692a72668c5c96e70fa1eb3ad9dd7
Sep 16 19:02:22.581358 ip-10-0-143-155 kubenswrapper[2578]: W0916 19:02:22.581334    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7335bfa6_a7d8_4560_b80d_1c18451f7fa2.slice/crio-89a1be98633c4ca0f21f6175ffb4882ca31b805101bfcfdae980745f0185144c WatchSource:0}: Error finding container 89a1be98633c4ca0f21f6175ffb4882ca31b805101bfcfdae980745f0185144c: Status 404 returned error can't find the container with id 89a1be98633c4ca0f21f6175ffb4882ca31b805101bfcfdae980745f0185144c
Sep 16 19:02:22.793890 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.793864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"
Sep 16 19:02:22.793890 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.793883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps" event={"ID":"a9f3751a-3c49-4069-9450-b124ab6343ef","Type":"ContainerDied","Data":"e3d992991e177d64d5e71c2a0864f98e8ba47b54bd46061c7e333492d2e4b635"}
Sep 16 19:02:22.794475 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.793956    2578 scope.go:117] "RemoveContainer" containerID="a3bc3539dc8d2b3e1fbcbce7fbc2e092dee00e05fc4b4c2cadc5c5bc81e35229"
Sep 16 19:02:22.795810 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.795755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" event={"ID":"7335bfa6-a7d8-4560-b80d-1c18451f7fa2","Type":"ContainerStarted","Data":"64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8"}
Sep 16 19:02:22.795945 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.795825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" event={"ID":"7335bfa6-a7d8-4560-b80d-1c18451f7fa2","Type":"ContainerStarted","Data":"89a1be98633c4ca0f21f6175ffb4882ca31b805101bfcfdae980745f0185144c"}
Sep 16 19:02:22.797663 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.797624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" event={"ID":"3abf83e4-61a6-45bb-8428-a47782b8c281","Type":"ContainerStarted","Data":"f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d"}
Sep 16 19:02:22.797781 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.797721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" event={"ID":"3abf83e4-61a6-45bb-8428-a47782b8c281","Type":"ContainerStarted","Data":"432abb7bc82bb31fe737f9079970bd76fbc692a72668c5c96e70fa1eb3ad9dd7"}
Sep 16 19:02:22.805452 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.805430    2578 scope.go:117] "RemoveContainer" containerID="e8399b9452ed243f0bc83d1d2904835ef13d48e2b2a1b350f5d5c42ba1a3addc"
Sep 16 19:02:22.814951 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.814909    2578 scope.go:117] "RemoveContainer" containerID="4164431e4985faac0a108aa1b834a5fd31cc52354438051e99f9c765a856b960"
Sep 16 19:02:22.877748 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.877656    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"]
Sep 16 19:02:22.884803 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:22.884768    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-gwkps"]
Sep 16 19:02:23.331381 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:23.331350    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f3751a-3c49-4069-9450-b124ab6343ef" path="/var/lib/kubelet/pods/a9f3751a-3c49-4069-9450-b124ab6343ef/volumes"
Sep 16 19:02:26.817889 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:26.817854    2578 generic.go:356] "Generic (PLEG): container finished" podID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerID="64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8" exitCode=0
Sep 16 19:02:26.818372 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:26.817927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" event={"ID":"7335bfa6-a7d8-4560-b80d-1c18451f7fa2","Type":"ContainerDied","Data":"64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8"}
Sep 16 19:02:26.819666 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:26.819647    2578 generic.go:356] "Generic (PLEG): container finished" podID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerID="f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d" exitCode=0
Sep 16 19:02:26.819757 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:26.819721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" event={"ID":"3abf83e4-61a6-45bb-8428-a47782b8c281","Type":"ContainerDied","Data":"f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d"}
Sep 16 19:02:27.827710 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.827675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" event={"ID":"7335bfa6-a7d8-4560-b80d-1c18451f7fa2","Type":"ContainerStarted","Data":"33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c"}
Sep 16 19:02:27.828209 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.827752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" event={"ID":"7335bfa6-a7d8-4560-b80d-1c18451f7fa2","Type":"ContainerStarted","Data":"c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7"}
Sep 16 19:02:27.828209 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.827973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:27.829908 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.829884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" event={"ID":"3abf83e4-61a6-45bb-8428-a47782b8c281","Type":"ContainerStarted","Data":"630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9"}
Sep 16 19:02:27.830039 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.829935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" event={"ID":"3abf83e4-61a6-45bb-8428-a47782b8c281","Type":"ContainerStarted","Data":"7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5"}
Sep 16 19:02:27.830148 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.830131    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:27.830192 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.830174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:27.831696 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.831670    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:02:27.850730 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.850664    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podStartSLOduration=2.96285258 podStartE2EDuration="6.850645729s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.887793149s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:27.848271392 +0000 UTC m=+1300.978818102" watchObservedRunningTime="2026-09-16 19:02:27.850645729 +0000 UTC m=+1300.981192441"
Sep 16 19:02:27.868723 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:27.868661    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podStartSLOduration=3.150648987 podStartE2EDuration="6.868645893s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717996906s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:27.867985908 +0000 UTC m=+1300.998532620" watchObservedRunningTime="2026-09-16 19:02:27.868645893 +0000 UTC m=+1300.999192603"
Sep 16 19:02:28.834089 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:28.834046    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:02:28.834089 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:28.834081    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:28.835132 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:28.835107    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:02:29.838021 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:29.837958    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:02:33.838177 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:33.838149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:02:33.838779 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:33.838750    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:02:34.843100 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:34.843067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:02:34.843762 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:34.843731    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:02:43.839323 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:43.839282    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:02:44.844069 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:44.844028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:02:53.839393 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:53.839352    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:02:54.844405 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:02:54.844366    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:03:03.839346 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:03.839300    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:04.843926 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:04.843886    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:03:13.839064 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:13.838956    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:14.844524 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:14.844485    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:03:23.838670 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:23.838632    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:24.844157 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:24.844117    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:03:33.839597 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:33.839550    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:34.844581 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:34.844537    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:03:43.839979 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:43.839948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:03:43.975065 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:43.975031    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"]
Sep 16 19:03:43.975311 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:43.975286    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container" containerID="cri-o://c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7" gracePeriod=30
Sep 16 19:03:43.975379 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:43.975336    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kube-rbac-proxy" containerID="cri-o://33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c" gracePeriod=30
Sep 16 19:03:43.976755 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:43.976734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:03:44.145762 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:44.145724    2578 generic.go:356] "Generic (PLEG): container finished" podID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerID="33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c" exitCode=2
Sep 16 19:03:44.145932 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:44.145797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" event={"ID":"7335bfa6-a7d8-4560-b80d-1c18451f7fa2","Type":"ContainerDied","Data":"33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c"}
Sep 16 19:03:44.839186 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:44.839133    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:03:49.020600 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.020574    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:03:49.061834 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.061799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf8hv\" (UniqueName: \"kubernetes.io/projected/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-api-access-gf8hv\") pod \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") "
Sep 16 19:03:49.062041 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.061903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-rbac-proxy-sar-config\") pod \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") "
Sep 16 19:03:49.062041 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.061945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-cabundle-cert\") pod \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") "
Sep 16 19:03:49.062185 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.062046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-proxy-tls\") pod \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") "
Sep 16 19:03:49.062185 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.062079    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kserve-provision-location\") pod \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\" (UID: \"7335bfa6-a7d8-4560-b80d-1c18451f7fa2\") "
Sep 16 19:03:49.062364 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.062334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7335bfa6-a7d8-4560-b80d-1c18451f7fa2" (UID: "7335bfa6-a7d8-4560-b80d-1c18451f7fa2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:49.062469 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.062344    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7335bfa6-a7d8-4560-b80d-1c18451f7fa2" (UID: "7335bfa6-a7d8-4560-b80d-1c18451f7fa2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:49.062469 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.062450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7335bfa6-a7d8-4560-b80d-1c18451f7fa2" (UID: "7335bfa6-a7d8-4560-b80d-1c18451f7fa2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:49.064274 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.064239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7335bfa6-a7d8-4560-b80d-1c18451f7fa2" (UID: "7335bfa6-a7d8-4560-b80d-1c18451f7fa2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:49.064274 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.064260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-api-access-gf8hv" (OuterVolumeSpecName: "kube-api-access-gf8hv") pod "7335bfa6-a7d8-4560-b80d-1c18451f7fa2" (UID: "7335bfa6-a7d8-4560-b80d-1c18451f7fa2"). InnerVolumeSpecName "kube-api-access-gf8hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:49.163474 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.163433    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:03:49.163474 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.163469    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:03:49.163474 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.163485    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:03:49.163747 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.163498    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:03:49.163747 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.163512    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf8hv\" (UniqueName: \"kubernetes.io/projected/7335bfa6-a7d8-4560-b80d-1c18451f7fa2-kube-api-access-gf8hv\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:03:49.169095 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.169066    2578 generic.go:356] "Generic (PLEG): container finished" podID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerID="c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7" exitCode=0
Sep 16 19:03:49.169218 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.169105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" event={"ID":"7335bfa6-a7d8-4560-b80d-1c18451f7fa2","Type":"ContainerDied","Data":"c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7"}
Sep 16 19:03:49.169218 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.169139    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"
Sep 16 19:03:49.169218 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.169181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv" event={"ID":"7335bfa6-a7d8-4560-b80d-1c18451f7fa2","Type":"ContainerDied","Data":"89a1be98633c4ca0f21f6175ffb4882ca31b805101bfcfdae980745f0185144c"}
Sep 16 19:03:49.169218 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.169209    2578 scope.go:117] "RemoveContainer" containerID="33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c"
Sep 16 19:03:49.177850 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.177829    2578 scope.go:117] "RemoveContainer" containerID="c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7"
Sep 16 19:03:49.185403 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.185384    2578 scope.go:117] "RemoveContainer" containerID="64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8"
Sep 16 19:03:49.193739 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.193719    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"]
Sep 16 19:03:49.193812 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.193745    2578 scope.go:117] "RemoveContainer" containerID="33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c"
Sep 16 19:03:49.194067 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:03:49.194047    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c\": container with ID starting with 33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c not found: ID does not exist" containerID="33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c"
Sep 16 19:03:49.194143 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.194075    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c"} err="failed to get container status \"33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c\": rpc error: code = NotFound desc = could not find container \"33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c\": container with ID starting with 33415e6f5b61ece99e137c1573f407e40a89931c8f5672fc338a5cfc5a484c2c not found: ID does not exist"
Sep 16 19:03:49.194143 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.194095    2578 scope.go:117] "RemoveContainer" containerID="c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7"
Sep 16 19:03:49.194337 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:03:49.194323    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7\": container with ID starting with c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7 not found: ID does not exist" containerID="c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7"
Sep 16 19:03:49.194380 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.194344    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7"} err="failed to get container status \"c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7\": rpc error: code = NotFound desc = could not find container \"c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7\": container with ID starting with c65b9b36868133af43313f39723eb8b713e3ea27e9103dfb72c41f95a23336c7 not found: ID does not exist"
Sep 16 19:03:49.194380 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.194356    2578 scope.go:117] "RemoveContainer" containerID="64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8"
Sep 16 19:03:49.194625 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:03:49.194606    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8\": container with ID starting with 64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8 not found: ID does not exist" containerID="64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8"
Sep 16 19:03:49.194680 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.194630    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8"} err="failed to get container status \"64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8\": rpc error: code = NotFound desc = could not find container \"64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8\": container with ID starting with 64fbfab8b007395b7777f27dd72780b1402c5fc69eef82aef1886b591b1aa8b8 not found: ID does not exist"
Sep 16 19:03:49.200585 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.200562    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-lmkzv"]
Sep 16 19:03:49.331824 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:03:49.331746    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" path="/var/lib/kubelet/pods/7335bfa6-a7d8-4560-b80d-1c18451f7fa2/volumes"
Sep 16 19:05:22.272066 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:22.272033    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"]
Sep 16 19:05:22.272813 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:22.272293    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" containerID="cri-o://7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5" gracePeriod=30
Sep 16 19:05:22.272813 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:22.272384    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kube-rbac-proxy" containerID="cri-o://630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9" gracePeriod=30
Sep 16 19:05:22.537660 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:22.537569    2578 generic.go:356] "Generic (PLEG): container finished" podID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerID="630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9" exitCode=2
Sep 16 19:05:22.537660 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:22.537645    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" event={"ID":"3abf83e4-61a6-45bb-8428-a47782b8c281","Type":"ContainerDied","Data":"630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9"}
Sep 16 19:05:23.834634 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:23.834590    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 19:05:23.838706 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:23.838684    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:05:27.125076 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.125054    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:05:27.210330 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.210240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-rbac-proxy-sar-config\") pod \"3abf83e4-61a6-45bb-8428-a47782b8c281\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") "
Sep 16 19:05:27.210330 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.210290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjjmk\" (UniqueName: \"kubernetes.io/projected/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-api-access-bjjmk\") pod \"3abf83e4-61a6-45bb-8428-a47782b8c281\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") "
Sep 16 19:05:27.210330 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.210325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abf83e4-61a6-45bb-8428-a47782b8c281-proxy-tls\") pod \"3abf83e4-61a6-45bb-8428-a47782b8c281\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") "
Sep 16 19:05:27.210627 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.210349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-cabundle-cert\") pod \"3abf83e4-61a6-45bb-8428-a47782b8c281\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") "
Sep 16 19:05:27.210627 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.210365    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abf83e4-61a6-45bb-8428-a47782b8c281-kserve-provision-location\") pod \"3abf83e4-61a6-45bb-8428-a47782b8c281\" (UID: \"3abf83e4-61a6-45bb-8428-a47782b8c281\") "
Sep 16 19:05:27.210735 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.210713    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3abf83e4-61a6-45bb-8428-a47782b8c281" (UID: "3abf83e4-61a6-45bb-8428-a47782b8c281"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:27.210783 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.210738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3abf83e4-61a6-45bb-8428-a47782b8c281" (UID: "3abf83e4-61a6-45bb-8428-a47782b8c281"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:27.210783 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.210757    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abf83e4-61a6-45bb-8428-a47782b8c281-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3abf83e4-61a6-45bb-8428-a47782b8c281" (UID: "3abf83e4-61a6-45bb-8428-a47782b8c281"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:27.212592 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.212573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3abf83e4-61a6-45bb-8428-a47782b8c281-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3abf83e4-61a6-45bb-8428-a47782b8c281" (UID: "3abf83e4-61a6-45bb-8428-a47782b8c281"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:27.212698 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.212655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-api-access-bjjmk" (OuterVolumeSpecName: "kube-api-access-bjjmk") pod "3abf83e4-61a6-45bb-8428-a47782b8c281" (UID: "3abf83e4-61a6-45bb-8428-a47782b8c281"). InnerVolumeSpecName "kube-api-access-bjjmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:27.311470 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.311429    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3abf83e4-61a6-45bb-8428-a47782b8c281-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:05:27.311470 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.311464    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:05:27.311470 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.311477    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3abf83e4-61a6-45bb-8428-a47782b8c281-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:05:27.311727 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.311489    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:05:27.311727 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.311503    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjjmk\" (UniqueName: \"kubernetes.io/projected/3abf83e4-61a6-45bb-8428-a47782b8c281-kube-api-access-bjjmk\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:05:27.558581 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.558549    2578 generic.go:356] "Generic (PLEG): container finished" podID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerID="7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5" exitCode=0
Sep 16 19:05:27.558815 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.558604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" event={"ID":"3abf83e4-61a6-45bb-8428-a47782b8c281","Type":"ContainerDied","Data":"7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5"}
Sep 16 19:05:27.558815 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.558615    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"
Sep 16 19:05:27.558815 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.558653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht" event={"ID":"3abf83e4-61a6-45bb-8428-a47782b8c281","Type":"ContainerDied","Data":"432abb7bc82bb31fe737f9079970bd76fbc692a72668c5c96e70fa1eb3ad9dd7"}
Sep 16 19:05:27.558815 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.558681    2578 scope.go:117] "RemoveContainer" containerID="630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9"
Sep 16 19:05:27.567998 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.567972    2578 scope.go:117] "RemoveContainer" containerID="7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5"
Sep 16 19:05:27.577776 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.577756    2578 scope.go:117] "RemoveContainer" containerID="f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d"
Sep 16 19:05:27.584566 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.584540    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"]
Sep 16 19:05:27.585836 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.585820    2578 scope.go:117] "RemoveContainer" containerID="630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9"
Sep 16 19:05:27.586273 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:05:27.586230    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9\": container with ID starting with 630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9 not found: ID does not exist" containerID="630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9"
Sep 16 19:05:27.586343 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.586303    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9"} err="failed to get container status \"630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9\": rpc error: code = NotFound desc = could not find container \"630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9\": container with ID starting with 630053e00ad725499f3ee679734c7b54cf5df855cc8b20ff23409d8221eb06b9 not found: ID does not exist"
Sep 16 19:05:27.586343 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.586328    2578 scope.go:117] "RemoveContainer" containerID="7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5"
Sep 16 19:05:27.586631 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:05:27.586616    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5\": container with ID starting with 7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5 not found: ID does not exist" containerID="7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5"
Sep 16 19:05:27.586674 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.586637    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5"} err="failed to get container status \"7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5\": rpc error: code = NotFound desc = could not find container \"7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5\": container with ID starting with 7371b78eb07efb65554c2bd17fe409aebe5731009d24d7b1be2d96634908cfc5 not found: ID does not exist"
Sep 16 19:05:27.586674 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.586651    2578 scope.go:117] "RemoveContainer" containerID="f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d"
Sep 16 19:05:27.586900 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:05:27.586884    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d\": container with ID starting with f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d not found: ID does not exist" containerID="f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d"
Sep 16 19:05:27.586943 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.586905    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d"} err="failed to get container status \"f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d\": rpc error: code = NotFound desc = could not find container \"f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d\": container with ID starting with f0e571424898b6f94e0bf514aff2634727fcf3704bd62ef553b74b769bcb084d not found: ID does not exist"
Sep 16 19:05:27.590603 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:27.590569    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-ff2ht"]
Sep 16 19:05:29.331202 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:29.331167    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" path="/var/lib/kubelet/pods/3abf83e4-61a6-45bb-8428-a47782b8c281/volumes"
Sep 16 19:05:36.515879 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.515843    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"]
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516143    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516157    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516165    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kube-rbac-proxy"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516170    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kube-rbac-proxy"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516186    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kube-rbac-proxy"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516192    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kube-rbac-proxy"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516199    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516204    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516211    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="storage-initializer"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516216    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="storage-initializer"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516222    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="storage-initializer"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516227    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="storage-initializer"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516272    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kube-rbac-proxy"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516279    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kube-rbac-proxy"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516284    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3abf83e4-61a6-45bb-8428-a47782b8c281" containerName="kserve-container"
Sep 16 19:05:36.516305 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.516291    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7335bfa6-a7d8-4560-b80d-1c18451f7fa2" containerName="kserve-container"
Sep 16 19:05:36.520725 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.520704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.523339 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.523315    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:05:36.523452 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.523355    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:05:36.523452 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.523373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkvz5\""
Sep 16 19:05:36.523452 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.523355    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 19:05:36.523452 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.523355    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 16 19:05:36.530470 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.530450    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"]
Sep 16 19:05:36.686659 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.686628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjkj\" (UniqueName: \"kubernetes.io/projected/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-api-access-rvjkj\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.686659 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.686660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.686893 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.686773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a20c3d52-d3bb-43b2-b185-6843028eda1a-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.788036 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.787922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a20c3d52-d3bb-43b2-b185-6843028eda1a-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.788036 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.787995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjkj\" (UniqueName: \"kubernetes.io/projected/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-api-access-rvjkj\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.788036 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.788039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.788618 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.788597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.790714 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.790679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a20c3d52-d3bb-43b2-b185-6843028eda1a-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.798693 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.798669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjkj\" (UniqueName: \"kubernetes.io/projected/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-api-access-rvjkj\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-x72wv\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:36.831771 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:36.831722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:37.109338 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:37.109196    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"]
Sep 16 19:05:37.111371 ip-10-0-143-155 kubenswrapper[2578]: W0916 19:05:37.111345    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20c3d52_d3bb_43b2_b185_6843028eda1a.slice/crio-d4563399acb6e876449e02ad83ebdb20c27aa64dcf190aabcb937e5b63e9dfdc WatchSource:0}: Error finding container d4563399acb6e876449e02ad83ebdb20c27aa64dcf190aabcb937e5b63e9dfdc: Status 404 returned error can't find the container with id d4563399acb6e876449e02ad83ebdb20c27aa64dcf190aabcb937e5b63e9dfdc
Sep 16 19:05:37.599230 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:37.599193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" event={"ID":"a20c3d52-d3bb-43b2-b185-6843028eda1a","Type":"ContainerStarted","Data":"f89e3cb30c71868805a41a9ee7339d3ec45d5a37b702c211568ad0a451c3cff8"}
Sep 16 19:05:37.599659 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:37.599254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" event={"ID":"a20c3d52-d3bb-43b2-b185-6843028eda1a","Type":"ContainerStarted","Data":"c5a5ab30a50b017a4f6ec70a42d36fc054ab4b5cffa314098cb361a6d4b7c9f9"}
Sep 16 19:05:37.599659 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:37.599279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" event={"ID":"a20c3d52-d3bb-43b2-b185-6843028eda1a","Type":"ContainerStarted","Data":"d4563399acb6e876449e02ad83ebdb20c27aa64dcf190aabcb937e5b63e9dfdc"}
Sep 16 19:05:37.599659 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:37.599315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:37.599659 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:37.599344    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:37.601526 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:37.601504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:37.624816 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:37.624760    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" podStartSLOduration=1.624740186 podStartE2EDuration="1.624740186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:37.622612693 +0000 UTC m=+1490.753159440" watchObservedRunningTime="2026-09-16 19:05:37.624740186 +0000 UTC m=+1490.755286899"
Sep 16 19:05:44.612434 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:44.612403    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:05:46.561639 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.561605    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"]
Sep 16 19:05:46.569385 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.569362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.572225 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.572199    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 16 19:05:46.572225 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.572211    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 19:05:46.572225 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.572209    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:05:46.572444 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.572211    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:05:46.577711 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.577690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"]
Sep 16 19:05:46.668774 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.668743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.668964 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.668791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6330c41c-29f0-4fb3-b471-4c96466be695-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.668964 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.668812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.668964 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.668922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6330c41c-29f0-4fb3-b471-4c96466be695-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.668964 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.668959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gbm\" (UniqueName: \"kubernetes.io/projected/6330c41c-29f0-4fb3-b471-4c96466be695-kube-api-access-d6gbm\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.770158 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.770115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.770340 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.770169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6330c41c-29f0-4fb3-b471-4c96466be695-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.770340 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.770282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.770340 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.770336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6330c41c-29f0-4fb3-b471-4c96466be695-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.770515 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.770361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gbm\" (UniqueName: \"kubernetes.io/projected/6330c41c-29f0-4fb3-b471-4c96466be695-kube-api-access-d6gbm\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.770802 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.770778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6330c41c-29f0-4fb3-b471-4c96466be695-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.770899 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.770825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.770952 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.770894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.772704 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.772675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6330c41c-29f0-4fb3-b471-4c96466be695-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.781892 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.781870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gbm\" (UniqueName: \"kubernetes.io/projected/6330c41c-29f0-4fb3-b471-4c96466be695-kube-api-access-d6gbm\") pod \"isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:46.880615 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:46.880531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:47.165377 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:47.165286    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"]
Sep 16 19:05:47.168817 ip-10-0-143-155 kubenswrapper[2578]: W0916 19:05:47.168786    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6330c41c_29f0_4fb3_b471_4c96466be695.slice/crio-2f533b3ea7bedab8a34a59ed482641191359f1f8bf293b4c9932aa2e7a47eab2 WatchSource:0}: Error finding container 2f533b3ea7bedab8a34a59ed482641191359f1f8bf293b4c9932aa2e7a47eab2: Status 404 returned error can't find the container with id 2f533b3ea7bedab8a34a59ed482641191359f1f8bf293b4c9932aa2e7a47eab2
Sep 16 19:05:47.323741 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:47.323710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 19:05:47.328622 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:47.328597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 19:05:47.643703 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:47.643663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerStarted","Data":"0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a"}
Sep 16 19:05:47.644119 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:47.643747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerStarted","Data":"2f533b3ea7bedab8a34a59ed482641191359f1f8bf293b4c9932aa2e7a47eab2"}
Sep 16 19:05:51.660334 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:51.660299    2578 generic.go:356] "Generic (PLEG): container finished" podID="6330c41c-29f0-4fb3-b471-4c96466be695" containerID="0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a" exitCode=0
Sep 16 19:05:51.660722 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:51.660373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerDied","Data":"0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a"}
Sep 16 19:05:52.668326 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.668292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerStarted","Data":"5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f"}
Sep 16 19:05:52.668807 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.668375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerStarted","Data":"96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603"}
Sep 16 19:05:52.668807 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.668402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerStarted","Data":"748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3"}
Sep 16 19:05:52.668807 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.668730    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:52.668807 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.668787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:52.668966 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.668816    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:52.670397 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.670364    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:05:52.671170 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.671143    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:05:52.692307 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:52.692255    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podStartSLOduration=3.011490266 podStartE2EDuration="6.692236626s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68074636s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:52.69166333 +0000 UTC m=+1505.822210041" watchObservedRunningTime="2026-09-16 19:05:52.692236626 +0000 UTC m=+1505.822783337"
Sep 16 19:05:53.672652 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:53.672610    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:05:53.673126 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:53.672986    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:05:58.677261 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:58.677226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:05:58.677909 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:58.677876    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:05:58.678104 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:05:58.678078    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:06:08.678763 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:08.678676    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:06:08.679188 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:08.679102    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:06:18.677984 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:18.677929    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:06:18.680375 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:18.678336    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:06:28.678275 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:28.678222    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:06:28.678726 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:28.678705    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:06:38.678524 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:38.678482    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:06:38.679028 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:38.678969    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:06:48.678672 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:48.678628    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:06:48.679179 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:48.679150    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:06:58.678308 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:58.678265    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:06:58.678877 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:06:58.678782    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:08.679119 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:08.679089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:07:08.679615 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:08.679180    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:07:21.597790 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:21.597760    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-x72wv_a20c3d52-d3bb-43b2-b185-6843028eda1a/kserve-container/0.log"
Sep 16 19:07:21.862337 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:21.862249    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"]
Sep 16 19:07:21.862541 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:21.862505    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" containerID="cri-o://748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3" gracePeriod=30
Sep 16 19:07:21.862671 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:21.862561    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy" containerID="cri-o://96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603" gracePeriod=30
Sep 16 19:07:21.862671 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:21.862518    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" containerID="cri-o://5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f" gracePeriod=30
Sep 16 19:07:21.910108 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:21.910068    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"]
Sep 16 19:07:21.910331 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:21.910294    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerName="kserve-container" containerID="cri-o://c5a5ab30a50b017a4f6ec70a42d36fc054ab4b5cffa314098cb361a6d4b7c9f9" gracePeriod=30
Sep 16 19:07:21.910392 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:21.910351    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerName="kube-rbac-proxy" containerID="cri-o://f89e3cb30c71868805a41a9ee7339d3ec45d5a37b702c211568ad0a451c3cff8" gracePeriod=30
Sep 16 19:07:22.043828 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.043793    2578 generic.go:356] "Generic (PLEG): container finished" podID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerID="f89e3cb30c71868805a41a9ee7339d3ec45d5a37b702c211568ad0a451c3cff8" exitCode=2
Sep 16 19:07:22.043828 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.043817    2578 generic.go:356] "Generic (PLEG): container finished" podID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerID="c5a5ab30a50b017a4f6ec70a42d36fc054ab4b5cffa314098cb361a6d4b7c9f9" exitCode=2
Sep 16 19:07:22.044085 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.043861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" event={"ID":"a20c3d52-d3bb-43b2-b185-6843028eda1a","Type":"ContainerDied","Data":"f89e3cb30c71868805a41a9ee7339d3ec45d5a37b702c211568ad0a451c3cff8"}
Sep 16 19:07:22.044085 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.043925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" event={"ID":"a20c3d52-d3bb-43b2-b185-6843028eda1a","Type":"ContainerDied","Data":"c5a5ab30a50b017a4f6ec70a42d36fc054ab4b5cffa314098cb361a6d4b7c9f9"}
Sep 16 19:07:22.046640 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.046610    2578 generic.go:356] "Generic (PLEG): container finished" podID="6330c41c-29f0-4fb3-b471-4c96466be695" containerID="96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603" exitCode=2
Sep 16 19:07:22.046760 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.046653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerDied","Data":"96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603"}
Sep 16 19:07:22.163196 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.163171    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:07:22.193538 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.193504    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-rbac-proxy-sar-config\") pod \"a20c3d52-d3bb-43b2-b185-6843028eda1a\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") "
Sep 16 19:07:22.193711 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.193560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a20c3d52-d3bb-43b2-b185-6843028eda1a-proxy-tls\") pod \"a20c3d52-d3bb-43b2-b185-6843028eda1a\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") "
Sep 16 19:07:22.193711 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.193592    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvjkj\" (UniqueName: \"kubernetes.io/projected/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-api-access-rvjkj\") pod \"a20c3d52-d3bb-43b2-b185-6843028eda1a\" (UID: \"a20c3d52-d3bb-43b2-b185-6843028eda1a\") "
Sep 16 19:07:22.193997 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.193958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a20c3d52-d3bb-43b2-b185-6843028eda1a" (UID: "a20c3d52-d3bb-43b2-b185-6843028eda1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:22.195902 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.195873    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20c3d52-d3bb-43b2-b185-6843028eda1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a20c3d52-d3bb-43b2-b185-6843028eda1a" (UID: "a20c3d52-d3bb-43b2-b185-6843028eda1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:22.195986 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.195878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-api-access-rvjkj" (OuterVolumeSpecName: "kube-api-access-rvjkj") pod "a20c3d52-d3bb-43b2-b185-6843028eda1a" (UID: "a20c3d52-d3bb-43b2-b185-6843028eda1a"). InnerVolumeSpecName "kube-api-access-rvjkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:22.295067 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.295002    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:07:22.295067 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.295067    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a20c3d52-d3bb-43b2-b185-6843028eda1a-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:07:22.295273 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:22.295081    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjkj\" (UniqueName: \"kubernetes.io/projected/a20c3d52-d3bb-43b2-b185-6843028eda1a-kube-api-access-rvjkj\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:07:23.052528 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:23.052499    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"
Sep 16 19:07:23.052528 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:23.052501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv" event={"ID":"a20c3d52-d3bb-43b2-b185-6843028eda1a","Type":"ContainerDied","Data":"d4563399acb6e876449e02ad83ebdb20c27aa64dcf190aabcb937e5b63e9dfdc"}
Sep 16 19:07:23.053066 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:23.052580    2578 scope.go:117] "RemoveContainer" containerID="f89e3cb30c71868805a41a9ee7339d3ec45d5a37b702c211568ad0a451c3cff8"
Sep 16 19:07:23.061590 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:23.061569    2578 scope.go:117] "RemoveContainer" containerID="c5a5ab30a50b017a4f6ec70a42d36fc054ab4b5cffa314098cb361a6d4b7c9f9"
Sep 16 19:07:23.083751 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:23.083715    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"]
Sep 16 19:07:23.091023 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:23.090982    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-x72wv"]
Sep 16 19:07:23.331558 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:23.331524    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" path="/var/lib/kubelet/pods/a20c3d52-d3bb-43b2-b185-6843028eda1a/volumes"
Sep 16 19:07:23.673775 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:23.673670    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:07:27.073777 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:27.073680    2578 generic.go:356] "Generic (PLEG): container finished" podID="6330c41c-29f0-4fb3-b471-4c96466be695" containerID="748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3" exitCode=0
Sep 16 19:07:27.073777 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:27.073758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerDied","Data":"748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3"}
Sep 16 19:07:28.673462 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:28.673419    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:07:28.678209 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:28.678175    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:07:28.678554 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:28.678530    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:33.673217 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:33.673169    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:07:33.673664 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:33.673335    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:07:38.673648 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:38.673551    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:07:38.678087 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:38.678052    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:07:38.678459 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:38.678430    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:39.054019 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:39.053984    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:39.088911 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:39.088875    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:39.103388 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:39.103365    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:39.119134 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:39.119105    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:39.994142 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:39.994106    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:40.018616 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:40.018584    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:40.034804 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:40.034774    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:40.059029 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:40.058992    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:40.983876 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:40.983846    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:41.004752 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:41.004726    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:41.020636 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:41.020610    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:41.037724 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:41.037689    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:41.908674 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:41.908643    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:41.925482 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:41.925456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:41.939863 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:41.939830    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:41.955184 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:41.955161    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:42.808199 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:42.808156    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:42.833720 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:42.833681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:42.845278 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:42.845248    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:42.859814 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:42.859786    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:43.673071 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:43.673032    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:07:43.734386 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:43.734359    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:43.753162 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:43.753134    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:43.768565 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:43.768542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:43.784738 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:43.784716    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:44.665931 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:44.665898    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:44.684058 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:44.684032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:44.700778 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:44.700746    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:44.718794 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:44.718770    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:45.606883 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:45.606851    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:45.631077 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:45.631036    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:45.653718 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:45.653690    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:45.677275 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:45.677225    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:46.558958 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:46.558927    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:46.582646 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:46.582612    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:46.609574 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:46.609551    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:46.633199 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:46.633164    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:47.550980 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:47.550949    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:47.577471 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:47.577440    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:47.606602 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:47.606571    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:47.634207 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:47.634178    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:48.496490 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.496459    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:48.512667 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.512637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:48.526823 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.526797    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:48.543850 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.543819    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:48.673188 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.673142    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 19:07:48.678030 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.677974    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:07:48.678191 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.678174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:07:48.678377 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.678361    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:48.678449 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:48.678438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:07:49.427113 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:49.427082    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:49.442802 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:49.442772    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:49.457528 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:49.457497    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:49.478348 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:49.478314    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:50.396089 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:50.396061    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:50.412373 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:50.412348    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:50.426186 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:50.426161    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:50.440267 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:50.440235    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:51.337785 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:51.337758    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kserve-container/0.log"
Sep 16 19:07:51.356040 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:51.355994    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/kube-rbac-proxy/0.log"
Sep 16 19:07:51.371874 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:51.371847    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/agent/0.log"
Sep 16 19:07:51.388405 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:51.388373    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7_6330c41c-29f0-4fb3-b471-4c96466be695/storage-initializer/0.log"
Sep 16 19:07:52.010764 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.010734    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:07:52.116118 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.116070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6330c41c-29f0-4fb3-b471-4c96466be695-kserve-provision-location\") pod \"6330c41c-29f0-4fb3-b471-4c96466be695\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") "
Sep 16 19:07:52.116118 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.116130    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-kube-rbac-proxy-sar-config\") pod \"6330c41c-29f0-4fb3-b471-4c96466be695\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") "
Sep 16 19:07:52.116432 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.116170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6gbm\" (UniqueName: \"kubernetes.io/projected/6330c41c-29f0-4fb3-b471-4c96466be695-kube-api-access-d6gbm\") pod \"6330c41c-29f0-4fb3-b471-4c96466be695\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") "
Sep 16 19:07:52.116432 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.116210    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6330c41c-29f0-4fb3-b471-4c96466be695-proxy-tls\") pod \"6330c41c-29f0-4fb3-b471-4c96466be695\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") "
Sep 16 19:07:52.116432 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.116243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-cabundle-cert\") pod \"6330c41c-29f0-4fb3-b471-4c96466be695\" (UID: \"6330c41c-29f0-4fb3-b471-4c96466be695\") "
Sep 16 19:07:52.116603 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.116467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6330c41c-29f0-4fb3-b471-4c96466be695-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6330c41c-29f0-4fb3-b471-4c96466be695" (UID: "6330c41c-29f0-4fb3-b471-4c96466be695"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:52.116603 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.116588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6330c41c-29f0-4fb3-b471-4c96466be695" (UID: "6330c41c-29f0-4fb3-b471-4c96466be695"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:52.116682 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.116645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6330c41c-29f0-4fb3-b471-4c96466be695" (UID: "6330c41c-29f0-4fb3-b471-4c96466be695"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:52.118532 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.118503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6330c41c-29f0-4fb3-b471-4c96466be695-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6330c41c-29f0-4fb3-b471-4c96466be695" (UID: "6330c41c-29f0-4fb3-b471-4c96466be695"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:52.118633 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.118540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6330c41c-29f0-4fb3-b471-4c96466be695-kube-api-access-d6gbm" (OuterVolumeSpecName: "kube-api-access-d6gbm") pod "6330c41c-29f0-4fb3-b471-4c96466be695" (UID: "6330c41c-29f0-4fb3-b471-4c96466be695"). InnerVolumeSpecName "kube-api-access-d6gbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:52.178562 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.178526    2578 generic.go:356] "Generic (PLEG): container finished" podID="6330c41c-29f0-4fb3-b471-4c96466be695" containerID="5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f" exitCode=0
Sep 16 19:07:52.178707 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.178587    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"
Sep 16 19:07:52.178707 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.178604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerDied","Data":"5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f"}
Sep 16 19:07:52.178707 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.178672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7" event={"ID":"6330c41c-29f0-4fb3-b471-4c96466be695","Type":"ContainerDied","Data":"2f533b3ea7bedab8a34a59ed482641191359f1f8bf293b4c9932aa2e7a47eab2"}
Sep 16 19:07:52.178707 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.178703    2578 scope.go:117] "RemoveContainer" containerID="5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f"
Sep 16 19:07:52.187627 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.187610    2578 scope.go:117] "RemoveContainer" containerID="96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603"
Sep 16 19:07:52.195248 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.195227    2578 scope.go:117] "RemoveContainer" containerID="748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3"
Sep 16 19:07:52.202945 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.202923    2578 scope.go:117] "RemoveContainer" containerID="0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a"
Sep 16 19:07:52.203797 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.203774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"]
Sep 16 19:07:52.209976 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.209952    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-58d9d7cffd-jqfr7"]
Sep 16 19:07:52.211416 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.211395    2578 scope.go:117] "RemoveContainer" containerID="5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f"
Sep 16 19:07:52.211719 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:07:52.211693    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f\": container with ID starting with 5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f not found: ID does not exist" containerID="5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f"
Sep 16 19:07:52.211771 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.211725    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f"} err="failed to get container status \"5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f\": rpc error: code = NotFound desc = could not find container \"5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f\": container with ID starting with 5ab7bcf24a9003544d10bede1f8c2fe9fae6419c234a1f901ee2a3404dac3b3f not found: ID does not exist"
Sep 16 19:07:52.211771 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.211748    2578 scope.go:117] "RemoveContainer" containerID="96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603"
Sep 16 19:07:52.212033 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:07:52.211997    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603\": container with ID starting with 96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603 not found: ID does not exist" containerID="96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603"
Sep 16 19:07:52.212082 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.212041    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603"} err="failed to get container status \"96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603\": rpc error: code = NotFound desc = could not find container \"96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603\": container with ID starting with 96807f0cbe5c6845d185ec1aad54d651f6498096da84b7d9f37e325bc72f8603 not found: ID does not exist"
Sep 16 19:07:52.212082 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.212057    2578 scope.go:117] "RemoveContainer" containerID="748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3"
Sep 16 19:07:52.212291 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:07:52.212276    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3\": container with ID starting with 748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3 not found: ID does not exist" containerID="748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3"
Sep 16 19:07:52.212335 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.212294    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3"} err="failed to get container status \"748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3\": rpc error: code = NotFound desc = could not find container \"748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3\": container with ID starting with 748f0cdacaeba8e632152187f975a4a289e976bf35b91e691eee8cdda4fb1ba3 not found: ID does not exist"
Sep 16 19:07:52.212335 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.212308    2578 scope.go:117] "RemoveContainer" containerID="0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a"
Sep 16 19:07:52.212591 ip-10-0-143-155 kubenswrapper[2578]: E0916 19:07:52.212565    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a\": container with ID starting with 0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a not found: ID does not exist" containerID="0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a"
Sep 16 19:07:52.212628 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.212601    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a"} err="failed to get container status \"0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a\": rpc error: code = NotFound desc = could not find container \"0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a\": container with ID starting with 0a094d2f01603860efefab3ecab70ee5b2ca4548bcc0fcb28ed183b7dbaa114a not found: ID does not exist"
Sep 16 19:07:52.216969 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.216953    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:07:52.217035 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.216972    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6gbm\" (UniqueName: \"kubernetes.io/projected/6330c41c-29f0-4fb3-b471-4c96466be695-kube-api-access-d6gbm\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:07:52.217035 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.216981    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6330c41c-29f0-4fb3-b471-4c96466be695-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:07:52.217035 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.216990    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6330c41c-29f0-4fb3-b471-4c96466be695-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:07:52.217035 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:52.216998    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6330c41c-29f0-4fb3-b471-4c96466be695-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Sep 16 19:07:53.331342 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:53.331306    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" path="/var/lib/kubelet/pods/6330c41c-29f0-4fb3-b471-4c96466be695/volumes"
Sep 16 19:07:57.599723 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:57.599689    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s25jm_310f9f3d-4268-4c5b-b046-afa2cebb3685/global-pull-secret-syncer/0.log"
Sep 16 19:07:57.684919 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:57.684885    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8tlk8_af08d1a0-b107-4438-b7ed-8061c969ba8d/konnectivity-agent/0.log"
Sep 16 19:07:57.802888 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:07:57.802860    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-155.ec2.internal_7b899a3c315c0814e264cff323a4907d/haproxy/0.log"
Sep 16 19:08:03.018083 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:03.018041    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzh5b_11420469-fbd0-4e40-91a1-8c649fd1765c/node-exporter/0.log"
Sep 16 19:08:03.062336 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:03.062311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzh5b_11420469-fbd0-4e40-91a1-8c649fd1765c/kube-rbac-proxy/0.log"
Sep 16 19:08:03.102269 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:03.102241    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzh5b_11420469-fbd0-4e40-91a1-8c649fd1765c/init-textfile/0.log"
Sep 16 19:08:05.043985 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.043953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-kx2rr_e75dc3ff-9a16-4a71-bb1b-a7193a872c73/networking-console-plugin/0.log"
Sep 16 19:08:05.546482 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.546451    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/1.log"
Sep 16 19:08:05.556418 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.556387    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-575r8_e7e40482-bb13-4380-8909-3dfc3a6fb77d/console-operator/2.log"
Sep 16 19:08:05.878195 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878101    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"]
Sep 16 19:08:05.878506 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878486    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerName="kube-rbac-proxy"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878508    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerName="kube-rbac-proxy"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878523    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="storage-initializer"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878531    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="storage-initializer"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878549    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878558    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878571    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878580    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878597    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy"
Sep 16 19:08:05.878604 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878606    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy"
Sep 16 19:08:05.879086 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878616    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerName="kserve-container"
Sep 16 19:08:05.879086 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878623    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerName="kserve-container"
Sep 16 19:08:05.879086 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878706    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerName="kube-rbac-proxy"
Sep 16 19:08:05.879086 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878720    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kube-rbac-proxy"
Sep 16 19:08:05.879086 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878728    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="agent"
Sep 16 19:08:05.879086 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878737    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6330c41c-29f0-4fb3-b471-4c96466be695" containerName="kserve-container"
Sep 16 19:08:05.879086 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.878745    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20c3d52-d3bb-43b2-b185-6843028eda1a" containerName="kserve-container"
Sep 16 19:08:05.883329 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.883307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:05.885775 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.885755    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gvsp\"/\"kube-root-ca.crt\""
Sep 16 19:08:05.887031 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.886992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9gvsp\"/\"default-dockercfg-lwr2v\""
Sep 16 19:08:05.887091 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.886992    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gvsp\"/\"openshift-service-ca.crt\""
Sep 16 19:08:05.891367 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.891348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"]
Sep 16 19:08:05.918535 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.918505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-proc\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:05.918535 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.918540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-podres\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:05.918745 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.918569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zgc\" (UniqueName: \"kubernetes.io/projected/9abe1031-ef98-4f2a-b54d-19cb3853a409-kube-api-access-n7zgc\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:05.918745 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.918660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-sys\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:05.918745 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:05.918711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-lib-modules\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.019674 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zgc\" (UniqueName: \"kubernetes.io/projected/9abe1031-ef98-4f2a-b54d-19cb3853a409-kube-api-access-n7zgc\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.019874 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-sys\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.019874 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-lib-modules\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.019874 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-proc\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.019874 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-podres\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.019874 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-sys\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.019874 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-proc\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.020154 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-lib-modules\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.020154 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.019900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9abe1031-ef98-4f2a-b54d-19cb3853a409-podres\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.029149 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.029122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zgc\" (UniqueName: \"kubernetes.io/projected/9abe1031-ef98-4f2a-b54d-19cb3853a409-kube-api-access-n7zgc\") pod \"perf-node-gather-daemonset-hxlrz\" (UID: \"9abe1031-ef98-4f2a-b54d-19cb3853a409\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.194076 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.193956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:06.481098 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:06.480968    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"]
Sep 16 19:08:06.483876 ip-10-0-143-155 kubenswrapper[2578]: W0916 19:08:06.483836    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9abe1031_ef98_4f2a_b54d_19cb3853a409.slice/crio-1fb1333199355a20e7423bc1b04c272441537e69130dcbc696b5199cd16eca8a WatchSource:0}: Error finding container 1fb1333199355a20e7423bc1b04c272441537e69130dcbc696b5199cd16eca8a: Status 404 returned error can't find the container with id 1fb1333199355a20e7423bc1b04c272441537e69130dcbc696b5199cd16eca8a
Sep 16 19:08:07.237267 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:07.237232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz" event={"ID":"9abe1031-ef98-4f2a-b54d-19cb3853a409","Type":"ContainerStarted","Data":"658c8ac8fe0c9bbd0cc75bb9ad3aef04a4b76d0c4ee1fc7566b683bd3dbc0115"}
Sep 16 19:08:07.237636 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:07.237312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz" event={"ID":"9abe1031-ef98-4f2a-b54d-19cb3853a409","Type":"ContainerStarted","Data":"1fb1333199355a20e7423bc1b04c272441537e69130dcbc696b5199cd16eca8a"}
Sep 16 19:08:07.237636 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:07.237437    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:07.260395 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:07.260343    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz" podStartSLOduration=2.260326521 podStartE2EDuration="2.260326521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:07.259228737 +0000 UTC m=+1640.389775449" watchObservedRunningTime="2026-09-16 19:08:07.260326521 +0000 UTC m=+1640.390873232"
Sep 16 19:08:07.787908 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:07.787879    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6rvpd_4209b046-f8e0-49d9-b964-32593cd91d30/dns/0.log"
Sep 16 19:08:07.817124 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:07.817091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6rvpd_4209b046-f8e0-49d9-b964-32593cd91d30/kube-rbac-proxy/0.log"
Sep 16 19:08:07.852505 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:07.852476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drqxf_b708bf47-0a07-4e28-b464-487766a7377c/dns-node-resolver/0.log"
Sep 16 19:08:08.486045 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:08.485996    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2cpl_68880f21-301d-4f7f-bb64-1f210f499111/node-ca/0.log"
Sep 16 19:08:09.730409 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:09.730384    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h9vd7_7089aa2c-4247-42fd-902e-83400da24150/serve-healthcheck-canary/0.log"
Sep 16 19:08:10.207935 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:10.207852    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mzxqq_6863d054-55d0-4434-a64f-0d4b323b29a8/kube-rbac-proxy/0.log"
Sep 16 19:08:10.236244 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:10.236212    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mzxqq_6863d054-55d0-4434-a64f-0d4b323b29a8/exporter/0.log"
Sep 16 19:08:10.265815 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:10.265784    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mzxqq_6863d054-55d0-4434-a64f-0d4b323b29a8/extractor/0.log"
Sep 16 19:08:12.964948 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:12.964914    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-n8xsh_952661c3-9ea2-4382-a8d0-f09a5147e19c/server/0.log"
Sep 16 19:08:13.160187 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:13.160149    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-88c82_a6f6d643-53b3-4782-a1f4-087b0a972fa0/manager/0.log"
Sep 16 19:08:13.252566 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:13.252538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-hxlrz"
Sep 16 19:08:13.323685 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:13.323649    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-jxpjh_14dde8ef-ab27-47bd-b0af-c619bfabb0a3/seaweedfs/0.log"
Sep 16 19:08:18.165046 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:18.164993    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vrrxx_6e8f7212-ea24-454f-af67-6961fd575bf4/migrator/0.log"
Sep 16 19:08:18.192851 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:18.192815    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vrrxx_6e8f7212-ea24-454f-af67-6961fd575bf4/graceful-termination/0.log"
Sep 16 19:08:19.934145 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:19.934115    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xx92x_2fdcb25e-2aad-4ca5-bca2-120ea66a478c/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:08:19.962156 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:19.962126    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xx92x_2fdcb25e-2aad-4ca5-bca2-120ea66a478c/egress-router-binary-copy/0.log"
Sep 16 19:08:19.989140 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:19.989112    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xx92x_2fdcb25e-2aad-4ca5-bca2-120ea66a478c/cni-plugins/0.log"
Sep 16 19:08:20.017698 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:20.017656    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xx92x_2fdcb25e-2aad-4ca5-bca2-120ea66a478c/bond-cni-plugin/0.log"
Sep 16 19:08:20.045622 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:20.045591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xx92x_2fdcb25e-2aad-4ca5-bca2-120ea66a478c/routeoverride-cni/0.log"
Sep 16 19:08:20.074857 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:20.074825    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xx92x_2fdcb25e-2aad-4ca5-bca2-120ea66a478c/whereabouts-cni-bincopy/0.log"
Sep 16 19:08:20.102756 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:20.102730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xx92x_2fdcb25e-2aad-4ca5-bca2-120ea66a478c/whereabouts-cni/0.log"
Sep 16 19:08:20.223813 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:20.223784    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t8sp4_f9b72d1b-d4e0-4edc-9fb8-2afeec39c5f5/kube-multus/0.log"
Sep 16 19:08:20.365149 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:20.365118    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wdr9l_b19e5fc5-c807-4afb-a848-7cc107eb74a6/network-metrics-daemon/0.log"
Sep 16 19:08:20.391230 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:20.391202    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wdr9l_b19e5fc5-c807-4afb-a848-7cc107eb74a6/kube-rbac-proxy/0.log"
Sep 16 19:08:21.206658 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:21.206627    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jg58_87a8beac-18a0-4743-9368-e0eea3507ad8/ovn-controller/0.log"
Sep 16 19:08:21.230571 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:21.230545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jg58_87a8beac-18a0-4743-9368-e0eea3507ad8/ovn-acl-logging/0.log"
Sep 16 19:08:21.266993 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:21.266964    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jg58_87a8beac-18a0-4743-9368-e0eea3507ad8/kube-rbac-proxy-node/0.log"
Sep 16 19:08:21.299560 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:21.299525    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jg58_87a8beac-18a0-4743-9368-e0eea3507ad8/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:08:21.327587 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:21.327560    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jg58_87a8beac-18a0-4743-9368-e0eea3507ad8/northd/0.log"
Sep 16 19:08:21.357484 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:21.357458    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jg58_87a8beac-18a0-4743-9368-e0eea3507ad8/nbdb/0.log"
Sep 16 19:08:21.388568 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:21.388539    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jg58_87a8beac-18a0-4743-9368-e0eea3507ad8/sbdb/0.log"
Sep 16 19:08:21.452272 ip-10-0-143-155 kubenswrapper[2578]: I0916 19:08:21.452246    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6jg58_87a8beac-18a0-4743-9368-e0eea3507ad8/ovnkube-controller/0.log"