Sep 18 20:20:21.643925 ip-10-0-138-158 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 20:20:25.056272 ip-10-0-138-158 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:25.056272 ip-10-0-138-158 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 20:20:25.056272 ip-10-0-138-158 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:25.056272 ip-10-0-138-158 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 20:20:25.056272 ip-10-0-138-158 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:25.056987 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.056344    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 20:20:25.059867 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059849    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:25.059867 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059866    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059870    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059873    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059875    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059878    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059881    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059884    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059887    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059889    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059893    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059895    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059902    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059905    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059908    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059910    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059913    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059916    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059918    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059921    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059923    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:25.059930 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059926    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059929    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059931    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059935    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059937    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059940    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059944    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059947    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059949    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059952    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059955    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059957    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059960    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059964    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059968    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059971    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059974    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059977    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059980    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:25.060406 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059983    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059985    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059988    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059990    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059993    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059995    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.059998    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060001    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060003    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060006    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060008    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060011    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060013    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060017    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060019    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060023    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060026    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060029    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060031    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060034    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060037    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:25.060886 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060039    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060042    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060045    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060047    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060050    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060052    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060055    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060058    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060060    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060063    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060067    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060069    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060072    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060074    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060077    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060079    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060082    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060084    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060087    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060089    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:25.061386 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060092    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060095    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060097    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060099    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060103    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060107    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060112    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060116    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060119    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060121    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060124    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060126    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060129    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060133    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060136    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060139    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060142    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060145    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060858    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:25.061877 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060865    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060869    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060872    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060875    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060878    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060881    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060884    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060887    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060890    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060893    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060896    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060898    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060901    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060904    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060907    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060909    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060912    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060915    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060917    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060920    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:25.062331 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060923    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060926    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060929    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060932    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060934    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060937    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060940    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060943    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060945    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060948    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060950    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060953    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060956    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060958    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060961    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060963    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060966    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060969    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060972    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060975    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:25.062848 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060978    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060980    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060983    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060985    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060988    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060993    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.060997    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061000    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061003    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061005    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061008    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061011    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061015    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061018    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061021    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061023    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061026    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061028    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061031    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:25.063340 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061034    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061036    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061039    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061041    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061044    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061047    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061049    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061052    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061054    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061058    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061060    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061063    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061066    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061069    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061072    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061074    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061077    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061080    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061082    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061086    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:25.063821 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061090    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061093    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061095    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061098    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061101    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061104    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061107    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061110    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061112    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061115    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061118    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061120    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061127    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061131    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061134    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061137    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061140    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061143    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061146    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061548    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:25.064330 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061555    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061558    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061563    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061566    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061569    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061572    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061575    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061578    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061581    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061584    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061586    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061589    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061592    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061594    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061597    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061599    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061602    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061605    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061607    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:25.064843 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061610    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061613    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061615    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061618    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061620    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061625    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061628    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061630    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061633    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061636    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061638    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061641    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061643    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061646    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061648    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061650    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061653    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061656    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061658    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:25.065297 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061661    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061663    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061666    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061669    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061671    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061674    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061676    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061679    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061681    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061684    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061686    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061689    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061691    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061694    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061696    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061699    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061702    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061704    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061708    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061711    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:25.065797 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061713    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061716    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061718    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061721    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061723    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061726    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061728    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061731    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061733    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061735    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061738    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061741    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061744    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061746    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061749    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061752    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061754    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061758    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061760    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:25.066274 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061763    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061765    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061768    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061771    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061775    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061780    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061783    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061786    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061788    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061791    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061794    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061798    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061802    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061805    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061808    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061810    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061813    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061815    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061818    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061820    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:25.066763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.061823    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061894    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061937    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061946    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061950    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061955    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061959    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061964    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061968    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061971    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061975    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061978    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061981    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061985    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061988    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061990    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061993    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061996    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.061999    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062003    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062006    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062009    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062012    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062018    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 20:20:25.067270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062021    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062024    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062027    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062031    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062034    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062037    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062040    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062043    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062046    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062050    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062053    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062056    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062059    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062062    2577 flags.go:64] FLAG: --enable-server="true"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062065    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062070    2577 flags.go:64] FLAG: --event-burst="100"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062073    2577 flags.go:64] FLAG: --event-qps="50"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062076    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062079    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062082    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062086    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062089    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062092    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062095    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062098    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 20:20:25.067859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062101    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062105    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062108    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062111    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062114    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062117    2577 flags.go:64] FLAG: --feature-gates=""
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062122    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062125    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062128    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062131    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062135    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062138    2577 flags.go:64] FLAG: --help="false"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062140    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062144    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062147    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062149    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062153    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062157    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062159    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062162    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062165    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062170    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062173    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062177    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 20:20:25.068511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062179    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062182    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062185    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062188    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062191    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062194    2577 flags.go:64] FLAG: --lock-file=""
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062197    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062200    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062203    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062208    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062211    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062213    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062216    2577 flags.go:64] FLAG: --logging-format="text"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062219    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062224    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062227    2577 flags.go:64] FLAG: --manifest-url=""
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062230    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062238    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062241    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062245    2577 flags.go:64] FLAG: --max-pods="110"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062247    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062251    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062253    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062256    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062260    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 20:20:25.069102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062263    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062266    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062273    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062276    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062280    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062283    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062289    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062295    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062298    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062301    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062304    2577 flags.go:64] FLAG: --port="10250"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062307    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062310    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06846fceff834ee40"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062313    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062316    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062319    2577 flags.go:64] FLAG: --register-node="true"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062322    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062325    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062328    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062331    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062334    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062338    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062342    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062345    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062348    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 20:20:25.069749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062351    2577 flags.go:64] FLAG: --runonce="false"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062354    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062357    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062360    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062363    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062366    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062368    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062371    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062374    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062377    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062380    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062383    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062387    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062391    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062395    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062398    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062403    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062406    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062409    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062413    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062417    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062420    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062423    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062425    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062428    2577 flags.go:64] FLAG: --v="2"
Sep 18 20:20:25.070360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062433    2577 flags.go:64] FLAG: --version="false"
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062437    2577 flags.go:64] FLAG: --vmodule=""
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062441    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062444    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062571    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062575    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062578    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062582    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062585    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062588    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062591    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062595    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062603    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062608    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062612    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062617    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062620    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062623    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062626    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062630    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:25.071039 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062633    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062636    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062639    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062642    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062645    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062647    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062650    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062653    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062655    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062658    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062660    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062663    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062666    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062668    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062671    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062674    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062677    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062679    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062682    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062684    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:25.071541 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062687    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062691    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062695    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062698    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062702    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062706    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062708    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062711    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062715    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062718    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062721    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062725    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062728    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062731    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062733    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062736    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062739    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062741    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:25.072026 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062744    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062747    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062749    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062752    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062755    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062758    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062760    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062763    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062765    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062768    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062771    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062773    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062776    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062778    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062781    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062784    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062787    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062789    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062792    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062794    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:25.072464 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062797    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062800    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062802    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062805    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062807    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062811    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062815    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062818    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062821    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062823    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062826    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062829    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062831    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062834    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062837    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062839    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062842    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062844    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062847    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062849    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:25.073015 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062852    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062855    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062858    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062861    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.062863    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.062869    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.069987    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.070005    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.070025    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.070034    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070092    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070098    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070102    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070105    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:25.073570 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070108    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070112    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070116    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070119    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070122    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070125    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070128    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070130    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070133    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070136    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070138    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070141    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070143    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070146    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070149    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070151    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070154    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070156    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070159    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:25.073922 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070162    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070165    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070167    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070170    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070172    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070175    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070180    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070182    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070185    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070188    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070190    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070193    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070196    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070199    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070201    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070203    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070207    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070211    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070214    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070218    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:25.074427 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070221    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070224    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070227    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070229    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070232    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070235    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070237    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070240    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070243    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070246    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070250    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070253    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070256    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070258    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070261    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070264    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070266    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070269    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070273    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070277    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:25.074933 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070279    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070282    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070285    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070287    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070290    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070293    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070295    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070298    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070301    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070303    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070306    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070308    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070311    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070313    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070316    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070319    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070321    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070324    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070327    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070329    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:25.075466 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070332    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070334    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070337    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070339    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070342    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070346    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070348    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070351    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070353    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070356    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070359    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070362    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070365    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070368    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070370    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:25.075980 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070373    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.070379    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070510    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070515    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070519    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070522    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070525    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070528    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070530    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070533    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070536    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070538    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070541    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070544    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070547    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:25.076368 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070549    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070552    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070555    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070557    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070560    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070563    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070565    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070568    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070571    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070573    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070576    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070579    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070581    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070584    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070587    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070590    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070593    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070596    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070605    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070610    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:25.076755 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070613    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070616    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070618    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070621    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070625    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070629    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070632    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070635    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070638    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070641    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070643    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070646    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070649    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070652    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070655    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070657    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070660    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070663    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070665    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:25.077249 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070668    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070671    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070674    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070677    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070679    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070682    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070685    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070687    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070690    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070694    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070696    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070699    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070702    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070704    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070707    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070709    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070712    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070715    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070717    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070720    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:25.077716 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070723    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070725    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070729    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070732    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070736    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070738    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070741    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070743    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070746    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070748    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070751    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070754    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070756    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070759    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070762    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070764    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070767    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070770    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070772    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:25.078400 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070775    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070778    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070782    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070786    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070789    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070792    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070795    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:25.070797    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.070802    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.070933    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.073052    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.073884    2577 server.go:1019] "Starting client certificate rotation"
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.073985    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 20:20:25.079268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.074016    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 20:20:25.079671 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.076235    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 20:20:25.080013 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.079993    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 20:20:25.088988 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.088962    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 18 20:20:25.090668 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.090654    2577 log.go:25] "Validated CRI v1 image API"
Sep 18 20:20:25.091526 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.091508    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 20:20:25.094521 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.094500    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c188aeae-9b9e-4eb4-b4ce-0b757d00171b:/dev/nvme0n1p3 c2a4afa2-3540-4f92-827b-1750ba098349:/dev/nvme0n1p4]
Sep 18 20:20:25.094572 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.094520    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 20:20:25.098731 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.098610    2577 manager.go:216] Machine: {Timestamp:2026-09-18 20:20:25.098109475 +0000 UTC m=+1.907895408 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3153257 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b0f42ff1a67b98d56991876f9e16c SystemUUID:ec2b0f42-ff1a-67b9-8d56-991876f9e16c BootID:34f717fa-9f48-4780-9eea-56ca073a71da Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:c4:58:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:c4:58:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:28:20:e6:ee:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 20:20:25.098731 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.098724    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 20:20:25.098845 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.098814    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 20:20:25.100353 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.100321    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 20:20:25.100511 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.100354    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 20:20:25.101605 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.101595    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 20:20:25.101645 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.101609    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 20:20:25.101645 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.101618    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 20:20:25.101645 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.101629    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 20:20:25.101726 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.101705    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 20:20:25.106520 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.106507    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 20:20:25.106656 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.106645    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 20:20:25.106694 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.106689    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 20:20:25.106722 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.106711    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 20:20:25.106753 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.106737    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 20:20:25.106753 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.106752    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 18 20:20:25.106808 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.106765    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 20:20:25.107192 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.107180    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 20:20:25.107237 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.107199    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 20:20:25.109118 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.109093    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 20:20:25.111619 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.111603    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 20:20:25.111689 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.111640    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 20:20:25.112898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112884    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 20:20:25.112945 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112906    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 20:20:25.112945 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112913    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 20:20:25.112945 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112919    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 20:20:25.112945 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112928    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 20:20:25.112945 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112934    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 20:20:25.113092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112951    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 20:20:25.113092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112960    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 20:20:25.113092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112970    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 20:20:25.113092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112976    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 20:20:25.113092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112993    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 20:20:25.113092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.112999    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 20:20:25.113092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.113025    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 20:20:25.113092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.113033    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 20:20:25.115391 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.115362    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 20:20:25.115508 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.115396    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 20:20:25.118912 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.118892    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-158.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 20:20:25.121804 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.121790    2577 server.go:1268] "Started kubelet"
Sep 18 20:20:25.121954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.121897    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 20:20:25.122073 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.121978    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 20:20:25.122347 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.121868    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 20:20:25.122446 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.122385    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 20:20:25.122696 ip-10-0-138-158 systemd[1]: Started Kubernetes Kubelet.
Sep 18 20:20:25.148035 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.148009    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 20:20:25.148137 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.148061    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qgkpr"
Sep 18 20:20:25.149070 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.149043    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 20:20:25.149420 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.149382    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 20:20:25.153071 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.153052    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qgkpr"
Sep 18 20:20:25.154198 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.153384    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-158.ec2.internal.18d683ce8f94ef7f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-158.ec2.internal,UID:ip-10-0-138-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-158.ec2.internal,},FirstTimestamp:2026-09-18 20:20:25.121755007 +0000 UTC m=+1.931540940,LastTimestamp:2026-09-18 20:20:25.121755007 +0000 UTC m=+1.931540940,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-158.ec2.internal,}"
Sep 18 20:20:25.157601 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.157579    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 20:20:25.157601 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.157588    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 20:20:25.157732 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.157671    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.158139 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.158120    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 20:20:25.158139 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.158122    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 20:20:25.158276 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.158149    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 20:20:25.158276 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.158238    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 20:20:25.158276 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.158249    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 20:20:25.159727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.159156    2577 factory.go:55] Registering systemd factory
Sep 18 20:20:25.159727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.159175    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 18 20:20:25.159727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.159373    2577 factory.go:153] Registering CRI-O factory
Sep 18 20:20:25.159727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.159383    2577 factory.go:223] Registration of the crio container factory successfully
Sep 18 20:20:25.159727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.159437    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 20:20:25.159727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.159455    2577 factory.go:103] Registering Raw factory
Sep 18 20:20:25.159727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.159470    2577 manager.go:1196] Started watching for new ooms in manager
Sep 18 20:20:25.160070 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.159896    2577 manager.go:349] Starting recovery of all containers
Sep 18 20:20:25.167303 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.167271    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:25.168820 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.168804    2577 manager.go:354] Recovery completed
Sep 18 20:20:25.169755 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.169736    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-158.ec2.internal\" not found" node="ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.173268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.173256    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:25.177801 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.177787    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:25.177876 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.177815    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:25.177876 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.177824    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:25.178395 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.178382    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 20:20:25.178395 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.178393    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:25.178530 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.178394    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 20:20:25.178530 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.178415    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 20:20:25.179255 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.179241    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:25.179315 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.179265    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:25.179315 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.179275    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:25.180614 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.180600    2577 policy_none.go:49] "None policy: Start"
Sep 18 20:20:25.180674 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.180618    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 20:20:25.180674 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.180631    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 20:20:25.182669 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.182657    2577 policy_none.go:47] "Start"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.224231    2577 manager.go:337] "Starting Device Plugin manager"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.224276    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.224290    2577 server.go:87] "Starting device plugin registration server"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.226197    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.226256    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.226464    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.226764    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.226772    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.228154    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.228608    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.236043    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.237191    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.237231    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.237255    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.237293    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 20:20:25.240390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.239445    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:25.327166 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.327135    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:25.328337 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.328321    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:25.328416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.328351    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:25.328416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.328362    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:25.328416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.328387    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.336006 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.335983    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.336107 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.336015    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-158.ec2.internal\": node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.337648 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.337631    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"]
Sep 18 20:20:25.337720 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.337708    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:25.339248 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.339233    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:25.339322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.339266    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:25.339322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.339282    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:25.341691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.341676    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:25.341837 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.341818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.341894 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.341857    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:25.342414 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.342389    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:25.342488 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.342401    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:25.342488 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.342449    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:25.342488 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.342458    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:25.342488 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.342429    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:25.342645 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.342508    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:25.344711 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.344697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.344789 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.344722    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:25.348661 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.348643    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:25.348757 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.348677    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:25.348757 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.348690    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:25.359116 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.359098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbbc3bde7a229372ed6b4ba2eb294ca7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"fbbc3bde7a229372ed6b4ba2eb294ca7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.359200 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.359123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbbc3bde7a229372ed6b4ba2eb294ca7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"fbbc3bde7a229372ed6b4ba2eb294ca7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.359200 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.359139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9544b8965977bc8ce7945a3e09bc38b4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-158.ec2.internal\" (UID: \"9544b8965977bc8ce7945a3e09bc38b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.359200 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.359178    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.361048 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.361034    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-158.ec2.internal\" not found" node="ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.366991 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.366976    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-158.ec2.internal\" not found" node="ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.459272 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.459239    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.459272 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.459261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbbc3bde7a229372ed6b4ba2eb294ca7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"fbbc3bde7a229372ed6b4ba2eb294ca7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.459502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.459284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbbc3bde7a229372ed6b4ba2eb294ca7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"fbbc3bde7a229372ed6b4ba2eb294ca7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.459502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.459301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9544b8965977bc8ce7945a3e09bc38b4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-158.ec2.internal\" (UID: \"9544b8965977bc8ce7945a3e09bc38b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.459502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.459335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9544b8965977bc8ce7945a3e09bc38b4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-158.ec2.internal\" (UID: \"9544b8965977bc8ce7945a3e09bc38b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.459502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.459336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbbc3bde7a229372ed6b4ba2eb294ca7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"fbbc3bde7a229372ed6b4ba2eb294ca7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.459502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.459363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbbc3bde7a229372ed6b4ba2eb294ca7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"fbbc3bde7a229372ed6b4ba2eb294ca7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.560236 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.560195    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.660344 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.660265    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.665442 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.665421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.670908 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.670892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.760604 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.760564    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.860987 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:25.860950    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Sep 18 20:20:25.927992 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.927920    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:25.958629 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.958584    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.969435 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.969410    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 20:20:25.971090 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.971075    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Sep 18 20:20:25.982625 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:25.982605    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 20:20:26.074075 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.074044    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 20:20:26.074516 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.074216    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 20:20:26.074516 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.074237    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 20:20:26.074516 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.074237    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 20:20:26.107362 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.107333    2577 apiserver.go:52] "Watching apiserver"
Sep 18 20:20:26.122814 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.122785    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 20:20:26.123133 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.123114    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-b8ng8","kube-system/konnectivity-agent-g44d8","kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp","openshift-dns/node-resolver-kpgfw","openshift-image-registry/node-ca-gzrh4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal","openshift-multus/network-metrics-daemon-rhttn","openshift-cluster-node-tuning-operator/tuned-q7jwp","openshift-multus/multus-additional-cni-plugins-v2nh2","openshift-multus/multus-c9w4s","openshift-network-diagnostics/network-check-target-hptqd","openshift-network-operator/iptables-alerter-bsrdj","openshift-ovn-kubernetes/ovnkube-node-dklfc"]
Sep 18 20:20:26.125572 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.125554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.125662 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.125641    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:26.127700 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.127684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:26.129892 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.129870    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 20:20:26.129988 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.129916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.129988 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.129878    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 20:20:26.129988 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.129882    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j9ndn\""
Sep 18 20:20:26.132454 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.132427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.132788 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.132771    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 20:20:26.132851 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.132813    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8hw2t\""
Sep 18 20:20:26.133048 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.133032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 20:20:26.133114 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.133101    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 20:20:26.134562 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.134545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.134664 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.134647    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hm8sg\""
Sep 18 20:20:26.134730 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.134698    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 20:20:26.134870 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.134856    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 20:20:26.136663 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.136611    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 20:20:26.136887 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.136866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:26.136928 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.136897    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 20:20:26.136967 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.136878    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 20:20:26.136967 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.136941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:26.137057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.136945    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vk2x9\""
Sep 18 20:20:26.139029 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.139014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.141211 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.141191    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8q7w8\""
Sep 18 20:20:26.141311 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.141249    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:20:26.141311 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.141293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.141679 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.141658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 20:20:26.143662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.143634    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 20:20:26.143757 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.143726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.143841 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.143823    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 20:20:26.143950 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.143935    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 20:20:26.144080 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.144061    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 20:20:26.144265 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.144252    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gdd7j\""
Sep 18 20:20:26.144310 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.144253    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 20:20:26.145873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.145853    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5gvkm\""
Sep 18 20:20:26.145984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.145863    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 20:20:26.146128 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.146113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:26.146204 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.146182    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:26.148234 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.148216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.150615 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.150579    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7wmfv\""
Sep 18 20:20:26.150682 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.150662    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 20:20:26.150735 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.150695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.150789 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.150739    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:20:26.151055 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.150895    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 20:20:26.152989 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.152972    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 20:20:26.153081 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.153064    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 20:20:26.153827 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.153812    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 20:20:26.153925 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.153909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 20:20:26.153925 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.153919    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 20:20:26.154037 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.154022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gcdtx\""
Sep 18 20:20:26.154296 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.154282    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 20:20:26.158428 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.158409    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 20:20:26.158936 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.158883    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 20:15:25 +0000 UTC" deadline="2028-06-29 13:39:54.777854018 +0000 UTC"
Sep 18 20:20:26.158983 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.158937    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15593h19m28.618919853s"
Sep 18 20:20:26.159189 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.159176    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 20:20:26.163029 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ncv\" (UniqueName: \"kubernetes.io/projected/f5c64d04-78fe-440d-81d1-b17abb3c02e0-kube-api-access-w4ncv\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.163200 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-system-cni-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.163267 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-systemd-units\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.163267 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-sys-fs\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.163267 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fa951d2-1593-4871-b3b3-76614e660378-serviceca\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.163416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrl8\" (UniqueName: \"kubernetes.io/projected/7fa951d2-1593-4871-b3b3-76614e660378-kube-api-access-hkrl8\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.163416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-cni-bin\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.163416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovnkube-script-lib\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.163416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-modprobe-d\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.163416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-conf-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-multus-certs\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244a261e-ab8c-41a4-9a04-09b22da8a488-tmp-dir\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27bf6b08-ab66-451f-bc9b-ce188564686c-dbus\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysconfig\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5c64d04-78fe-440d-81d1-b17abb3c02e0-tmp\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-slash\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-netns\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.163667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-cni-bin\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-socket-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtk28\" (UniqueName: \"kubernetes.io/projected/d77a339a-d325-4683-a6ae-945a5b0ebae1-kube-api-access-mtk28\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22f68618-1475-4194-99f3-cb41d4662db9-cni-binary-copy\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-etc-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnmcm\" (UniqueName: \"kubernetes.io/projected/009a3dff-41dc-4ee6-9d76-d36b1572b38c-kube-api-access-vnmcm\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-device-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-etc-selinux\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-lib-modules\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-var-lib-kubelet\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-cni-multus\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/244a261e-ab8c-41a4-9a04-09b22da8a488-hosts-file\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-socket-dir-parent\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22f68618-1475-4194-99f3-cb41d4662db9-multus-daemon-config\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfmt9\" (UniqueName: \"kubernetes.io/projected/244a261e-ab8c-41a4-9a04-09b22da8a488-kube-api-access-sfmt9\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.163999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9688a58-6f94-40c5-bb65-85b0fd7bf192-iptables-alerter-script\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.164020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-kubelet\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/296cdbc3-c56a-422c-bc96-74c175ddfcc6-konnectivity-ca\") pod \"konnectivity-agent-g44d8\" (UID: \"296cdbc3-c56a-422c-bc96-74c175ddfcc6\") " pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-systemd\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-run-ovn-kubernetes\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-cnibin\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-etc-kubernetes\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9688a58-6f94-40c5-bb65-85b0fd7bf192-host-slash\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvvq\" (UniqueName: \"kubernetes.io/projected/d9688a58-6f94-40c5-bb65-85b0fd7bf192-kube-api-access-dkvvq\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-ovn\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovnkube-config\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-registration-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6pc\" (UniqueName: \"kubernetes.io/projected/7795f7f8-da44-460a-a884-07d834c38ba3-kube-api-access-tj6pc\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysctl-d\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.164712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-k8s-cni-cncf-io\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-cni-netd\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-kubernetes\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysctl-conf\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-run-netns\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27bf6b08-ab66-451f-bc9b-ce188564686c-kubelet-config\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-run\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-host\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-kubelet\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-node-log\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-systemd\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-cni-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.165263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-cnibin\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-log-socket\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovn-node-metrics-cert\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ssm8\" (UniqueName: \"kubernetes.io/projected/96105b04-10dc-4cb6-a96d-ee837afa9e66-kube-api-access-2ssm8\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/296cdbc3-c56a-422c-bc96-74c175ddfcc6-agent-certs\") pod \"konnectivity-agent-g44d8\" (UID: \"296cdbc3-c56a-422c-bc96-74c175ddfcc6\") " pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-env-overrides\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fa951d2-1593-4871-b3b3-76614e660378-host\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-sys\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-system-cni-dir\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-hostroot\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.164997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsmh8\" (UniqueName: \"kubernetes.io/projected/22f68618-1475-4194-99f3-cb41d4662db9-kube-api-access-qsmh8\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.165036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-var-lib-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.165743 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.165072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-tuned\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.168390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.168366    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 20:20:26.193338 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.193267    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x57t8"
Sep 18 20:20:26.202031 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.202011    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x57t8"
Sep 18 20:20:26.265314 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrl8\" (UniqueName: \"kubernetes.io/projected/7fa951d2-1593-4871-b3b3-76614e660378-kube-api-access-hkrl8\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.265421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-cni-bin\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.265421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovnkube-script-lib\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.265421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-modprobe-d\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.265421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-conf-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.265421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-cni-bin\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-multus-certs\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244a261e-ab8c-41a4-9a04-09b22da8a488-tmp-dir\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27bf6b08-ab66-451f-bc9b-ce188564686c-dbus\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysconfig\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-modprobe-d\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5c64d04-78fe-440d-81d1-b17abb3c02e0-tmp\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-slash\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.265690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-netns\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-netns\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-slash\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265832    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-conf-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.265948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244a261e-ab8c-41a4-9a04-09b22da8a488-tmp-dir\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-cni-bin\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27bf6b08-ab66-451f-bc9b-ce188564686c-dbus\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-socket-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtk28\" (UniqueName: \"kubernetes.io/projected/d77a339a-d325-4683-a6ae-945a5b0ebae1-kube-api-access-mtk28\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22f68618-1475-4194-99f3-cb41d4662db9-cni-binary-copy\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.266110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-multus-certs\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-etc-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmcm\" (UniqueName: \"kubernetes.io/projected/009a3dff-41dc-4ee6-9d76-d36b1572b38c-kube-api-access-vnmcm\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-device-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.266186    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-device-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.266258    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret podName:27bf6b08-ab66-451f-bc9b-ce188564686c nodeName:}" failed. No retries permitted until 2026-09-18 20:20:26.766236159 +0000 UTC m=+3.576022079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret") pod "global-pull-secret-syncer-b8ng8" (UID: "27bf6b08-ab66-451f-bc9b-ce188564686c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-etc-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-cni-bin\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysconfig\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovnkube-script-lib\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-etc-selinux\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-socket-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-lib-modules\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-var-lib-kubelet\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-cni-multus\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/244a261e-ab8c-41a4-9a04-09b22da8a488-hosts-file\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.266641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-etc-selinux\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-socket-dir-parent\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-lib-modules\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-var-lib-kubelet\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22f68618-1475-4194-99f3-cb41d4662db9-multus-daemon-config\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-socket-dir-parent\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22f68618-1475-4194-99f3-cb41d4662db9-cni-binary-copy\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-cni-multus\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/244a261e-ab8c-41a4-9a04-09b22da8a488-hosts-file\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfmt9\" (UniqueName: \"kubernetes.io/projected/244a261e-ab8c-41a4-9a04-09b22da8a488-kube-api-access-sfmt9\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9688a58-6f94-40c5-bb65-85b0fd7bf192-iptables-alerter-script\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-kubelet\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/296cdbc3-c56a-422c-bc96-74c175ddfcc6-konnectivity-ca\") pod \"konnectivity-agent-g44d8\" (UID: \"296cdbc3-c56a-422c-bc96-74c175ddfcc6\") " pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-systemd\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.267438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-run-ovn-kubernetes\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-cnibin\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-etc-kubernetes\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9688a58-6f94-40c5-bb65-85b0fd7bf192-host-slash\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvvq\" (UniqueName: \"kubernetes.io/projected/d9688a58-6f94-40c5-bb65-85b0fd7bf192-kube-api-access-dkvvq\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-ovn\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-run-ovn-kubernetes\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-etc-kubernetes\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovnkube-config\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22f68618-1475-4194-99f3-cb41d4662db9-multus-daemon-config\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-registration-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.266880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-var-lib-kubelet\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-systemd\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-cnibin\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9688a58-6f94-40c5-bb65-85b0fd7bf192-host-slash\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6pc\" (UniqueName: \"kubernetes.io/projected/7795f7f8-da44-460a-a884-07d834c38ba3-kube-api-access-tj6pc\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:26.268245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-run-ovn\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysctl-d\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-registration-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-k8s-cni-cncf-io\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-cni-netd\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-kubernetes\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysctl-conf\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-run-netns\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27bf6b08-ab66-451f-bc9b-ce188564686c-kubelet-config\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/296cdbc3-c56a-422c-bc96-74c175ddfcc6-konnectivity-ca\") pod \"konnectivity-agent-g44d8\" (UID: \"296cdbc3-c56a-422c-bc96-74c175ddfcc6\") " pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-run\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9688a58-6f94-40c5-bb65-85b0fd7bf192-iptables-alerter-script\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-run\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-host\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27bf6b08-ab66-451f-bc9b-ce188564686c-kubelet-config\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.268979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-host-run-k8s-cni-cncf-io\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-host\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-cni-netd\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysctl-d\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-run-netns\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovnkube-config\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.267728    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-kubernetes\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-kubelet\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.267798    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs podName:7795f7f8-da44-460a-a884-07d834c38ba3 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:26.767781839 +0000 UTC m=+3.577567771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs") pod "network-metrics-daemon-rhttn" (UID: "7795f7f8-da44-460a-a884-07d834c38ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-sysctl-conf\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-kubelet\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-node-log\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.269855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-systemd\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-systemd\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-node-log\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-cni-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-cnibin\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.267970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-log-socket\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268010    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-log-socket\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-multus-cni-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-cnibin\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovn-node-metrics-cert\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssm8\" (UniqueName: \"kubernetes.io/projected/96105b04-10dc-4cb6-a96d-ee837afa9e66-kube-api-access-2ssm8\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/296cdbc3-c56a-422c-bc96-74c175ddfcc6-agent-certs\") pod \"konnectivity-agent-g44d8\" (UID: \"296cdbc3-c56a-422c-bc96-74c175ddfcc6\") " pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-env-overrides\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fa951d2-1593-4871-b3b3-76614e660378-host\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.270683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-sys\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-system-cni-dir\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-hostroot\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsmh8\" (UniqueName: \"kubernetes.io/projected/22f68618-1475-4194-99f3-cb41d4662db9-kube-api-access-qsmh8\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-var-lib-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-tuned\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ncv\" (UniqueName: \"kubernetes.io/projected/f5c64d04-78fe-440d-81d1-b17abb3c02e0-kube-api-access-w4ncv\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-system-cni-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-systemd-units\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-sys-fs\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fa951d2-1593-4871-b3b3-76614e660378-serviceca\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/009a3dff-41dc-4ee6-9d76-d36b1572b38c-env-overrides\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fa951d2-1593-4871-b3b3-76614e660378-host\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5c64d04-78fe-440d-81d1-b17abb3c02e0-sys\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-var-lib-openvswitch\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d77a339a-d325-4683-a6ae-945a5b0ebae1-system-cni-dir\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-hostroot\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.271286 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/009a3dff-41dc-4ee6-9d76-d36b1572b38c-systemd-units\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.271873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96105b04-10dc-4cb6-a96d-ee837afa9e66-sys-fs\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.271873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.268968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22f68618-1475-4194-99f3-cb41d4662db9-system-cni-dir\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.271873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.269243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d77a339a-d325-4683-a6ae-945a5b0ebae1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.271873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.269406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fa951d2-1593-4871-b3b3-76614e660378-serviceca\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.271873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.269698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5c64d04-78fe-440d-81d1-b17abb3c02e0-tmp\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.271873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.270435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/009a3dff-41dc-4ee6-9d76-d36b1572b38c-ovn-node-metrics-cert\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.271873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.270919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/296cdbc3-c56a-422c-bc96-74c175ddfcc6-agent-certs\") pod \"konnectivity-agent-g44d8\" (UID: \"296cdbc3-c56a-422c-bc96-74c175ddfcc6\") " pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:26.271873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.271027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f5c64d04-78fe-440d-81d1-b17abb3c02e0-etc-tuned\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.276535 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.276503    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:26.276622 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.276542    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:26.276622 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.276553    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft7fq for pod openshift-network-diagnostics/network-check-target-hptqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:26.276622 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.276619    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq podName:fa298501-12e5-49d3-9272-b6a6f045d362 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:26.776606606 +0000 UTC m=+3.586392525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ft7fq" (UniqueName: "kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq") pod "network-check-target-hptqd" (UID: "fa298501-12e5-49d3-9272-b6a6f045d362") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:26.279052 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.279031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtk28\" (UniqueName: \"kubernetes.io/projected/d77a339a-d325-4683-a6ae-945a5b0ebae1-kube-api-access-mtk28\") pod \"multus-additional-cni-plugins-v2nh2\" (UID: \"d77a339a-d325-4683-a6ae-945a5b0ebae1\") " pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.279145 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.279067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnmcm\" (UniqueName: \"kubernetes.io/projected/009a3dff-41dc-4ee6-9d76-d36b1572b38c-kube-api-access-vnmcm\") pod \"ovnkube-node-dklfc\" (UID: \"009a3dff-41dc-4ee6-9d76-d36b1572b38c\") " pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.279145 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.279034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfmt9\" (UniqueName: \"kubernetes.io/projected/244a261e-ab8c-41a4-9a04-09b22da8a488-kube-api-access-sfmt9\") pod \"node-resolver-kpgfw\" (UID: \"244a261e-ab8c-41a4-9a04-09b22da8a488\") " pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.279145 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.279116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrl8\" (UniqueName: \"kubernetes.io/projected/7fa951d2-1593-4871-b3b3-76614e660378-kube-api-access-hkrl8\") pod \"node-ca-gzrh4\" (UID: \"7fa951d2-1593-4871-b3b3-76614e660378\") " pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.282582 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.282542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ncv\" (UniqueName: \"kubernetes.io/projected/f5c64d04-78fe-440d-81d1-b17abb3c02e0-kube-api-access-w4ncv\") pod \"tuned-q7jwp\" (UID: \"f5c64d04-78fe-440d-81d1-b17abb3c02e0\") " pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.282582 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.282540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6pc\" (UniqueName: \"kubernetes.io/projected/7795f7f8-da44-460a-a884-07d834c38ba3-kube-api-access-tj6pc\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:26.282867 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.282849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvvq\" (UniqueName: \"kubernetes.io/projected/d9688a58-6f94-40c5-bb65-85b0fd7bf192-kube-api-access-dkvvq\") pod \"iptables-alerter-bsrdj\" (UID: \"d9688a58-6f94-40c5-bb65-85b0fd7bf192\") " pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.283000 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.282983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsmh8\" (UniqueName: \"kubernetes.io/projected/22f68618-1475-4194-99f3-cb41d4662db9-kube-api-access-qsmh8\") pod \"multus-c9w4s\" (UID: \"22f68618-1475-4194-99f3-cb41d4662db9\") " pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.283770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.283752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssm8\" (UniqueName: \"kubernetes.io/projected/96105b04-10dc-4cb6-a96d-ee837afa9e66-kube-api-access-2ssm8\") pod \"aws-ebs-csi-driver-node-szjnp\" (UID: \"96105b04-10dc-4cb6-a96d-ee837afa9e66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.283858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.283791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bsrdj"
Sep 18 20:20:26.288634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.288620    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:26.315820 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.315786    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9544b8965977bc8ce7945a3e09bc38b4.slice/crio-c49027e65781e69233b68f7f0ca0051b5269873053974be12cfed56d112fac52 WatchSource:0}: Error finding container c49027e65781e69233b68f7f0ca0051b5269873053974be12cfed56d112fac52: Status 404 returned error can't find the container with id c49027e65781e69233b68f7f0ca0051b5269873053974be12cfed56d112fac52
Sep 18 20:20:26.316315 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.316292    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009a3dff_41dc_4ee6_9d76_d36b1572b38c.slice/crio-4ff17cc9cac85773ce24610e0bad2a2a4215ae294e92643c3b57f467320d9c84 WatchSource:0}: Error finding container 4ff17cc9cac85773ce24610e0bad2a2a4215ae294e92643c3b57f467320d9c84: Status 404 returned error can't find the container with id 4ff17cc9cac85773ce24610e0bad2a2a4215ae294e92643c3b57f467320d9c84
Sep 18 20:20:26.317169 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.317153    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbc3bde7a229372ed6b4ba2eb294ca7.slice/crio-432fdfc8df8086c535ed73d8f7ad54d50b113ad149ff024de817209262e6c0d8 WatchSource:0}: Error finding container 432fdfc8df8086c535ed73d8f7ad54d50b113ad149ff024de817209262e6c0d8: Status 404 returned error can't find the container with id 432fdfc8df8086c535ed73d8f7ad54d50b113ad149ff024de817209262e6c0d8
Sep 18 20:20:26.317916 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.317901    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9688a58_6f94_40c5_bb65_85b0fd7bf192.slice/crio-1cafd8ba08aa916dcf0f21a1ebcfa9236bffa7932d72aa0081178d4a8c5e34c7 WatchSource:0}: Error finding container 1cafd8ba08aa916dcf0f21a1ebcfa9236bffa7932d72aa0081178d4a8c5e34c7: Status 404 returned error can't find the container with id 1cafd8ba08aa916dcf0f21a1ebcfa9236bffa7932d72aa0081178d4a8c5e34c7
Sep 18 20:20:26.320961 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.320943    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:20:26.470083 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.469996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:26.479061 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.479031    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296cdbc3_c56a_422c_bc96_74c175ddfcc6.slice/crio-09ca415c2f00c398f2a088038b10a9897ddfdd3a0ab630e34936bc230d559bf3 WatchSource:0}: Error finding container 09ca415c2f00c398f2a088038b10a9897ddfdd3a0ab630e34936bc230d559bf3: Status 404 returned error can't find the container with id 09ca415c2f00c398f2a088038b10a9897ddfdd3a0ab630e34936bc230d559bf3
Sep 18 20:20:26.498669 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.498648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp"
Sep 18 20:20:26.506381 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.506359    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96105b04_10dc_4cb6_a96d_ee837afa9e66.slice/crio-1b19c0e9bfb7bcbab20d6788674efeb9fc1b6a06ccfeaa930cf1bae04bf0ca62 WatchSource:0}: Error finding container 1b19c0e9bfb7bcbab20d6788674efeb9fc1b6a06ccfeaa930cf1bae04bf0ca62: Status 404 returned error can't find the container with id 1b19c0e9bfb7bcbab20d6788674efeb9fc1b6a06ccfeaa930cf1bae04bf0ca62
Sep 18 20:20:26.512345 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.512324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kpgfw"
Sep 18 20:20:26.520452 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.520424    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244a261e_ab8c_41a4_9a04_09b22da8a488.slice/crio-b8f50a8621094d3ec153c74af9633e3a9825515b3354c7463d4df14e0fe41b9e WatchSource:0}: Error finding container b8f50a8621094d3ec153c74af9633e3a9825515b3354c7463d4df14e0fe41b9e: Status 404 returned error can't find the container with id b8f50a8621094d3ec153c74af9633e3a9825515b3354c7463d4df14e0fe41b9e
Sep 18 20:20:26.526405 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.526385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gzrh4"
Sep 18 20:20:26.534593 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.534568    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa951d2_1593_4871_b3b3_76614e660378.slice/crio-fb4e129489bd86808cf29e87ac0b04e5170a958388ef2cfcbf0e81dbc50fefea WatchSource:0}: Error finding container fb4e129489bd86808cf29e87ac0b04e5170a958388ef2cfcbf0e81dbc50fefea: Status 404 returned error can't find the container with id fb4e129489bd86808cf29e87ac0b04e5170a958388ef2cfcbf0e81dbc50fefea
Sep 18 20:20:26.540764 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.540741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q7jwp"
Sep 18 20:20:26.546785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.546678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2nh2"
Sep 18 20:20:26.548928 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.548906    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c64d04_78fe_440d_81d1_b17abb3c02e0.slice/crio-29410c8b27ae7358f910c00eb95bec2254c172e5681840ac342ff8910d626dac WatchSource:0}: Error finding container 29410c8b27ae7358f910c00eb95bec2254c172e5681840ac342ff8910d626dac: Status 404 returned error can't find the container with id 29410c8b27ae7358f910c00eb95bec2254c172e5681840ac342ff8910d626dac
Sep 18 20:20:26.556855 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.556835    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77a339a_d325_4683_a6ae_945a5b0ebae1.slice/crio-254bb43f9823eb0540a6076d90aa699fbbdc7b2ee31fb3c5fcbcc5f162670841 WatchSource:0}: Error finding container 254bb43f9823eb0540a6076d90aa699fbbdc7b2ee31fb3c5fcbcc5f162670841: Status 404 returned error can't find the container with id 254bb43f9823eb0540a6076d90aa699fbbdc7b2ee31fb3c5fcbcc5f162670841
Sep 18 20:20:26.562590 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.562573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9w4s"
Sep 18 20:20:26.571508 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:26.571470    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f68618_1475_4194_99f3_cb41d4662db9.slice/crio-c97cb25f8f2e1fd9cbab5afe231d3053b81845aa275d2741fad0c86252ec43f3 WatchSource:0}: Error finding container c97cb25f8f2e1fd9cbab5afe231d3053b81845aa275d2741fad0c86252ec43f3: Status 404 returned error can't find the container with id c97cb25f8f2e1fd9cbab5afe231d3053b81845aa275d2741fad0c86252ec43f3
Sep 18 20:20:26.660615 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.660586    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:26.771685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.771470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:26.771685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.771574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:26.771685 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.771628    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:26.771959 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.771688    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:26.771959 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.771710    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs podName:7795f7f8-da44-460a-a884-07d834c38ba3 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:27.77168875 +0000 UTC m=+4.581474672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs") pod "network-metrics-daemon-rhttn" (UID: "7795f7f8-da44-460a-a884-07d834c38ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:26.771959 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.771739    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret podName:27bf6b08-ab66-451f-bc9b-ce188564686c nodeName:}" failed. No retries permitted until 2026-09-18 20:20:27.771724962 +0000 UTC m=+4.581510896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret") pod "global-pull-secret-syncer-b8ng8" (UID: "27bf6b08-ab66-451f-bc9b-ce188564686c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:26.872582 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:26.872520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:26.872799 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.872744    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:26.872799 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.872768    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:26.872799 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.872782    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft7fq for pod openshift-network-diagnostics/network-check-target-hptqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:26.872960 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:26.872840    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq podName:fa298501-12e5-49d3-9272-b6a6f045d362 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:27.872822549 +0000 UTC m=+4.682608475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft7fq" (UniqueName: "kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq") pod "network-check-target-hptqd" (UID: "fa298501-12e5-49d3-9272-b6a6f045d362") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:27.204644 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.204605    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 20:15:26 +0000 UTC" deadline="2028-02-15 00:09:49.87292859 +0000 UTC"
Sep 18 20:20:27.204644 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.204642    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12339h49m22.668290537s"
Sep 18 20:20:27.233098 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.233062    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:27.240661 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.240519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:27.240661 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.240641    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:27.241012 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.240996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:27.241120 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.241098    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:27.257853 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.257626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q7jwp" event={"ID":"f5c64d04-78fe-440d-81d1-b17abb3c02e0","Type":"ContainerStarted","Data":"29410c8b27ae7358f910c00eb95bec2254c172e5681840ac342ff8910d626dac"}
Sep 18 20:20:27.271716 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.271105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp" event={"ID":"96105b04-10dc-4cb6-a96d-ee837afa9e66","Type":"ContainerStarted","Data":"1b19c0e9bfb7bcbab20d6788674efeb9fc1b6a06ccfeaa930cf1bae04bf0ca62"}
Sep 18 20:20:27.283054 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.283018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"4ff17cc9cac85773ce24610e0bad2a2a4215ae294e92643c3b57f467320d9c84"}
Sep 18 20:20:27.309237 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.309193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal" event={"ID":"fbbc3bde7a229372ed6b4ba2eb294ca7","Type":"ContainerStarted","Data":"432fdfc8df8086c535ed73d8f7ad54d50b113ad149ff024de817209262e6c0d8"}
Sep 18 20:20:27.336270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.336222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal" event={"ID":"9544b8965977bc8ce7945a3e09bc38b4","Type":"ContainerStarted","Data":"c49027e65781e69233b68f7f0ca0051b5269873053974be12cfed56d112fac52"}
Sep 18 20:20:27.357221 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.357169    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9w4s" event={"ID":"22f68618-1475-4194-99f3-cb41d4662db9","Type":"ContainerStarted","Data":"c97cb25f8f2e1fd9cbab5afe231d3053b81845aa275d2741fad0c86252ec43f3"}
Sep 18 20:20:27.384335 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.384042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" event={"ID":"d77a339a-d325-4683-a6ae-945a5b0ebae1","Type":"ContainerStarted","Data":"254bb43f9823eb0540a6076d90aa699fbbdc7b2ee31fb3c5fcbcc5f162670841"}
Sep 18 20:20:27.391616 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.391294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gzrh4" event={"ID":"7fa951d2-1593-4871-b3b3-76614e660378","Type":"ContainerStarted","Data":"fb4e129489bd86808cf29e87ac0b04e5170a958388ef2cfcbf0e81dbc50fefea"}
Sep 18 20:20:27.426891 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.426649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kpgfw" event={"ID":"244a261e-ab8c-41a4-9a04-09b22da8a488","Type":"ContainerStarted","Data":"b8f50a8621094d3ec153c74af9633e3a9825515b3354c7463d4df14e0fe41b9e"}
Sep 18 20:20:27.461033 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.458508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g44d8" event={"ID":"296cdbc3-c56a-422c-bc96-74c175ddfcc6","Type":"ContainerStarted","Data":"09ca415c2f00c398f2a088038b10a9897ddfdd3a0ab630e34936bc230d559bf3"}
Sep 18 20:20:27.488570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.488528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bsrdj" event={"ID":"d9688a58-6f94-40c5-bb65-85b0fd7bf192","Type":"ContainerStarted","Data":"1cafd8ba08aa916dcf0f21a1ebcfa9236bffa7932d72aa0081178d4a8c5e34c7"}
Sep 18 20:20:27.637825 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.637766    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:27.781358 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.781272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:27.781358 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.781348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:27.781632 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.781436    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:27.781632 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.781489    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:27.781632 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.781533    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs podName:7795f7f8-da44-460a-a884-07d834c38ba3 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:29.781510354 +0000 UTC m=+6.591296297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs") pod "network-metrics-daemon-rhttn" (UID: "7795f7f8-da44-460a-a884-07d834c38ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:27.781632 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.781555    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret podName:27bf6b08-ab66-451f-bc9b-ce188564686c nodeName:}" failed. No retries permitted until 2026-09-18 20:20:29.781544453 +0000 UTC m=+6.591330376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret") pod "global-pull-secret-syncer-b8ng8" (UID: "27bf6b08-ab66-451f-bc9b-ce188564686c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:27.882250 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:27.882209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:27.882430 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.882412    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:27.882533 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.882443    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:27.882533 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.882455    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft7fq for pod openshift-network-diagnostics/network-check-target-hptqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:27.882645 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:27.882532    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq podName:fa298501-12e5-49d3-9272-b6a6f045d362 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:29.882513336 +0000 UTC m=+6.692299275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft7fq" (UniqueName: "kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq") pod "network-check-target-hptqd" (UID: "fa298501-12e5-49d3-9272-b6a6f045d362") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:28.199725 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:28.199177    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:28.205427 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:28.205368    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 20:15:26 +0000 UTC" deadline="2028-06-04 01:46:40.571859428 +0000 UTC"
Sep 18 20:20:28.205427 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:28.205426    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14981h26m12.366436588s"
Sep 18 20:20:28.238078 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:28.238047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:28.238241 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:28.238167    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:29.238103 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:29.238069    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:29.238577 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.238206    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:29.238783 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:29.238762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:29.238897 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.238875    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:29.799666 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:29.799398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:29.799666 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:29.799498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:29.799666 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.799582    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:29.799666 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.799613    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:29.799666 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.799661    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret podName:27bf6b08-ab66-451f-bc9b-ce188564686c nodeName:}" failed. No retries permitted until 2026-09-18 20:20:33.799639358 +0000 UTC m=+10.609425299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret") pod "global-pull-secret-syncer-b8ng8" (UID: "27bf6b08-ab66-451f-bc9b-ce188564686c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:29.799666 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.799681    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs podName:7795f7f8-da44-460a-a884-07d834c38ba3 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:33.799671193 +0000 UTC m=+10.609457120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs") pod "network-metrics-daemon-rhttn" (UID: "7795f7f8-da44-460a-a884-07d834c38ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:29.900788 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:29.900746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:29.900998 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.900981    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:29.901082 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.901008    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:29.901082 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.901021    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft7fq for pod openshift-network-diagnostics/network-check-target-hptqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:29.901192 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:29.901083    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq podName:fa298501-12e5-49d3-9272-b6a6f045d362 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:33.901066229 +0000 UTC m=+10.710852154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft7fq" (UniqueName: "kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq") pod "network-check-target-hptqd" (UID: "fa298501-12e5-49d3-9272-b6a6f045d362") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:30.238307 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:30.238269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:30.238871 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:30.238393    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:31.239249 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:31.238566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:31.239249 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:31.238685    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:31.239249 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:31.239047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:31.239249 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:31.239152    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:32.238363 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:32.238327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:32.238629 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:32.238466    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:33.238305 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:33.237680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:33.238305 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.237840    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:33.238305 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:33.237958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:33.238305 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.238160    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:33.834013 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:33.833817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:33.834013 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:33.833939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:33.834013 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.833992    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:33.834316 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.834090    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs podName:7795f7f8-da44-460a-a884-07d834c38ba3 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.834071573 +0000 UTC m=+18.643857497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs") pod "network-metrics-daemon-rhttn" (UID: "7795f7f8-da44-460a-a884-07d834c38ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:33.834316 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.834102    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:33.834316 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.834178    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret podName:27bf6b08-ab66-451f-bc9b-ce188564686c nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.834144929 +0000 UTC m=+18.643930852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret") pod "global-pull-secret-syncer-b8ng8" (UID: "27bf6b08-ab66-451f-bc9b-ce188564686c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:33.934626 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:33.934568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:33.934822 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.934764    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:33.934822 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.934789    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:33.934822 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.934802    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft7fq for pod openshift-network-diagnostics/network-check-target-hptqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:33.934995 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:33.934866    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq podName:fa298501-12e5-49d3-9272-b6a6f045d362 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.934847918 +0000 UTC m=+18.744633861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft7fq" (UniqueName: "kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq") pod "network-check-target-hptqd" (UID: "fa298501-12e5-49d3-9272-b6a6f045d362") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:34.237652 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:34.237561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:34.237815 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:34.237706    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:35.238172 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:35.238139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:35.238651 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:35.238260    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:35.238651 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:35.238314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:35.238651 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:35.238435    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:36.237883 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:36.237846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:36.238074 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:36.237991    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:37.238080 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:37.238041    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:37.238488 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:37.238178    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:37.238488 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:37.238218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:37.238488 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:37.238304    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:38.237967 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:38.237926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:38.238259 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:38.238052    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:39.238013 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:39.237979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:39.238215 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:39.238103    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:39.238215 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:39.238157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:39.238669 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:39.238268    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:40.237847 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:40.237812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:40.238000 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:40.237951    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:41.237956 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:41.237923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:41.238410 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:41.237926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:41.238410 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.238044    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:41.238410 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.238160    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:41.891171 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:41.891092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:41.891171 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:41.891164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:41.891414 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.891264    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:41.891414 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.891296    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:41.891414 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.891336    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs podName:7795f7f8-da44-460a-a884-07d834c38ba3 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:57.891316017 +0000 UTC m=+34.701101943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs") pod "network-metrics-daemon-rhttn" (UID: "7795f7f8-da44-460a-a884-07d834c38ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:41.891414 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.891362    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret podName:27bf6b08-ab66-451f-bc9b-ce188564686c nodeName:}" failed. No retries permitted until 2026-09-18 20:20:57.891344286 +0000 UTC m=+34.701130206 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret") pod "global-pull-secret-syncer-b8ng8" (UID: "27bf6b08-ab66-451f-bc9b-ce188564686c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:41.992038 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:41.992010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:41.992217 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.992180    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:41.992217 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.992203    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:41.992217 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.992218    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft7fq for pod openshift-network-diagnostics/network-check-target-hptqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:41.992354 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:41.992275    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq podName:fa298501-12e5-49d3-9272-b6a6f045d362 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:57.992260654 +0000 UTC m=+34.802046575 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft7fq" (UniqueName: "kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq") pod "network-check-target-hptqd" (UID: "fa298501-12e5-49d3-9272-b6a6f045d362") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:42.237563 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:42.237488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:42.237822 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:42.237626    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:43.238525 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:43.238469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:43.238525 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:43.238503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:43.239049 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:43.238610    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:43.239049 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:43.238713    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:44.237491 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:44.237445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:44.237662 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:44.237577    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:45.238439 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:45.238402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:45.238789 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:45.238522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:45.238789 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:45.238522    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:45.238789 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:45.238645    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:45.532582 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:45.532549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q7jwp" event={"ID":"f5c64d04-78fe-440d-81d1-b17abb3c02e0","Type":"ContainerStarted","Data":"882a954c8ed46f0d4d6e825b1781a6d9f7b230afa7889392a7929661bffba302"}
Sep 18 20:20:45.534068 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:45.534046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal" event={"ID":"9544b8965977bc8ce7945a3e09bc38b4","Type":"ContainerStarted","Data":"b1adbe59a5afc325e3d36e701e3cacf047f09fdd5478a43a846dcd169e220bf1"}
Sep 18 20:20:46.237816 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.237627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:46.237995 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:46.237897    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:46.538052 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.538023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kpgfw" event={"ID":"244a261e-ab8c-41a4-9a04-09b22da8a488","Type":"ContainerStarted","Data":"bf96bc4737e2524deb2c1f8e2006bf452fe000f953df554438a5bb44cac91785"}
Sep 18 20:20:46.539683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.539628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g44d8" event={"ID":"296cdbc3-c56a-422c-bc96-74c175ddfcc6","Type":"ContainerStarted","Data":"d71ac52ec5adc2f1919df708cc5533e2f3e79bff6218248a4fa1f994b6c2fa84"}
Sep 18 20:20:46.541092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.541071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bsrdj" event={"ID":"d9688a58-6f94-40c5-bb65-85b0fd7bf192","Type":"ContainerStarted","Data":"77bee9f3d97139b8faf61b329fbc0f0147f03d8854d03b2d87b427d4e2d72b40"}
Sep 18 20:20:46.542775 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.542755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp" event={"ID":"96105b04-10dc-4cb6-a96d-ee837afa9e66","Type":"ContainerStarted","Data":"7877905c2e7879f40fa09150f5ada513b75f34996a394eb33ce7acb0385459d3"}
Sep 18 20:20:46.545804 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.545778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"7a63182f40a73715bb203f9c9aa71fa4cea07bd3fcd901b7e31e8241c837e70d"}
Sep 18 20:20:46.545907 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.545856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"c34ea896c259305a124a87bd62661d1e665c4dc4d4e890b1ea1d7f4c1174191d"}
Sep 18 20:20:46.545907 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.545902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"b84f95c0643fb2b54bb171d9ece2312b7e461a808d79867ec8195581dafbcf58"}
Sep 18 20:20:46.546008 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.545949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"1d97edeb53e09fdd9c09bf389ba8d223007c82a982ebf1bbf49f60659b775916"}
Sep 18 20:20:46.546008 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.545997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"ec34e6c235478fb14624800f6acb9c22f86bd7d16a305fab2a0cd4fd4fe8405d"}
Sep 18 20:20:46.546107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.546042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"353d2993c6789bd8820ab875ffa4dff6e9aaf499793abaa788d520b6dba47ae1"}
Sep 18 20:20:46.547217 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.547196    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbbc3bde7a229372ed6b4ba2eb294ca7" containerID="699275b1d867d8b71875287a3d655738dc6cdfc5c215fa3be98607010979abd7" exitCode=0
Sep 18 20:20:46.547318 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.547261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal" event={"ID":"fbbc3bde7a229372ed6b4ba2eb294ca7","Type":"ContainerDied","Data":"699275b1d867d8b71875287a3d655738dc6cdfc5c215fa3be98607010979abd7"}
Sep 18 20:20:46.548862 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.548836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9w4s" event={"ID":"22f68618-1475-4194-99f3-cb41d4662db9","Type":"ContainerStarted","Data":"49a5c0e9d2ecbd393fbe915d4ce355e68f4cf528f4cee6e5fadd574cdd9ec226"}
Sep 18 20:20:46.552538 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.552504    2577 generic.go:356] "Generic (PLEG): container finished" podID="d77a339a-d325-4683-a6ae-945a5b0ebae1" containerID="79ef18b6395b0dd2ec980e66a620c453a4b70db56f1de173d0411841dfb5d102" exitCode=0
Sep 18 20:20:46.553552 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.553525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" event={"ID":"d77a339a-d325-4683-a6ae-945a5b0ebae1","Type":"ContainerDied","Data":"79ef18b6395b0dd2ec980e66a620c453a4b70db56f1de173d0411841dfb5d102"}
Sep 18 20:20:46.556566 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.556395    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal" podStartSLOduration=21.556379003 podStartE2EDuration="21.556379003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:45.552809718 +0000 UTC m=+22.362595660" watchObservedRunningTime="2026-09-18 20:20:46.556379003 +0000 UTC m=+23.366164948"
Sep 18 20:20:46.556650 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.556565    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kpgfw" podStartSLOduration=2.901244324 podStartE2EDuration="21.556555011s" totalImagesPullingTime="18.655310687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:46.556124646 +0000 UTC m=+23.365910589" watchObservedRunningTime="2026-09-18 20:20:46.556555011 +0000 UTC m=+23.366340954"
Sep 18 20:20:46.557179 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.557152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gzrh4" event={"ID":"7fa951d2-1593-4871-b3b3-76614e660378","Type":"ContainerStarted","Data":"834fd81f425b99738e70daa7782e2697537a1c81be05cf8d2bbabcc080d24d00"}
Sep 18 20:20:46.611998 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.611944    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g44d8" podStartSLOduration=2.915618945 podStartE2EDuration="21.611923728s" totalImagesPullingTime="18.696304783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:46.611530852 +0000 UTC m=+23.421316796" watchObservedRunningTime="2026-09-18 20:20:46.611923728 +0000 UTC m=+23.421709671"
Sep 18 20:20:46.631992 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.631827    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c9w4s" podStartSLOduration=2.96447101 podStartE2EDuration="21.631813505s" totalImagesPullingTime="18.667342495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:46.631678644 +0000 UTC m=+23.441464585" watchObservedRunningTime="2026-09-18 20:20:46.631813505 +0000 UTC m=+23.441599445"
Sep 18 20:20:46.648539 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.648502    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gzrh4" podStartSLOduration=2.95942738 podStartE2EDuration="21.648487578s" totalImagesPullingTime="18.689060198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:46.648167472 +0000 UTC m=+23.457953405" watchObservedRunningTime="2026-09-18 20:20:46.648487578 +0000 UTC m=+23.458273514"
Sep 18 20:20:46.674999 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.674960    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q7jwp" podStartSLOduration=3.001188221 podStartE2EDuration="21.674946617s" totalImagesPullingTime="18.673758396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:46.674699859 +0000 UTC m=+23.484485803" watchObservedRunningTime="2026-09-18 20:20:46.674946617 +0000 UTC m=+23.484732562"
Sep 18 20:20:46.960760 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:46.960734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:47.238080 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:47.238001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:47.238341 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:47.238004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:47.238341 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:47.238152    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:47.238341 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:47.238207    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:47.423873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:47.423735    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 20:20:47.563823 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:47.563738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal" event={"ID":"fbbc3bde7a229372ed6b4ba2eb294ca7","Type":"ContainerStarted","Data":"c72ed44042d95de4add2239363ad0b36418c9ff3871dc3a8fa640ee80dbc40f2"}
Sep 18 20:20:47.565917 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:47.565888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp" event={"ID":"96105b04-10dc-4cb6-a96d-ee837afa9e66","Type":"ContainerStarted","Data":"0248f69a4357d93e5448f0d0e2aef287530e34213e52896ca05c7118238574e1"}
Sep 18 20:20:47.581936 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:47.581884    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal" podStartSLOduration=22.581868573 podStartE2EDuration="22.581868573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:47.581800762 +0000 UTC m=+24.391586705" watchObservedRunningTime="2026-09-18 20:20:47.581868573 +0000 UTC m=+24.391654517"
Sep 18 20:20:47.601299 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:47.601251    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bsrdj" podStartSLOduration=3.693738519 podStartE2EDuration="22.601240503s" totalImagesPullingTime="18.907501984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:47.600889582 +0000 UTC m=+24.410675526" watchObservedRunningTime="2026-09-18 20:20:47.601240503 +0000 UTC m=+24.411026446"
Sep 18 20:20:48.238456 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:48.238410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:48.238661 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:48.238548    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:48.239005 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:48.238933    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T20:20:47.423755076Z","UUID":"eb43edda-3e6a-46e0-9095-6cc16c0d8c0e","Handler":null,"Name":"","Endpoint":""}
Sep 18 20:20:48.240870 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:48.240847    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 20:20:48.241005 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:48.240879    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 20:20:49.238125 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:49.238087    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:49.238125 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:49.238106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:49.238719 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:49.238228    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:49.238719 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:49.238359    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:49.576621 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:49.576514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp" event={"ID":"96105b04-10dc-4cb6-a96d-ee837afa9e66","Type":"ContainerStarted","Data":"1c771022ce926b088eea38ab7f2e5a90fd53481acec040d15767c9ccc94c6d54"}
Sep 18 20:20:49.580519 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:49.580446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"d97b7dc1f2ae373a97ba84ec4b87b8cf0d681cd3cd54ea27800b4e8c0ed82feb"}
Sep 18 20:20:49.596329 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:49.596267    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-szjnp" podStartSLOduration=3.528597175 podStartE2EDuration="24.596252419s" totalImagesPullingTime="21.067655244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:49.5961175 +0000 UTC m=+26.405903435" watchObservedRunningTime="2026-09-18 20:20:49.596252419 +0000 UTC m=+26.406038360"
Sep 18 20:20:50.238218 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:50.238186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:50.238677 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:50.238310    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:51.237678 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.237508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:51.237913 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.237517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:51.237913 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:51.237745    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:51.237913 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:51.237826    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:51.248118 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.248097    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:51.248658 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.248612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:51.585540 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.585510    2577 generic.go:356] "Generic (PLEG): container finished" podID="d77a339a-d325-4683-a6ae-945a5b0ebae1" containerID="784e9c2c36b672a351659c5d8dbe5ebea25472213572a4e450c048987c1c67a3" exitCode=0
Sep 18 20:20:51.585708 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.585597    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" event={"ID":"d77a339a-d325-4683-a6ae-945a5b0ebae1","Type":"ContainerDied","Data":"784e9c2c36b672a351659c5d8dbe5ebea25472213572a4e450c048987c1c67a3"}
Sep 18 20:20:51.589586 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.589557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" event={"ID":"009a3dff-41dc-4ee6-9d76-d36b1572b38c","Type":"ContainerStarted","Data":"70d0f8358702e56696adfb7b3f2c837b916c7017133923b265b7e832e526b2d0"}
Sep 18 20:20:51.590115 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.590098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g44d8"
Sep 18 20:20:51.726181 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.726135    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc" podStartSLOduration=7.194305494 podStartE2EDuration="26.726119888s" totalImagesPullingTime="19.531814394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:51.725877387 +0000 UTC m=+28.535663323" watchObservedRunningTime="2026-09-18 20:20:51.726119888 +0000 UTC m=+28.535905830"
Sep 18 20:20:51.795512 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.795489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:51.900029 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:51.899951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:52.237704 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.237637    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:52.237829 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:52.237741    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:52.595200 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.595109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:52.596533 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.596510    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:52.596631 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.596592    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:52.616627 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.616590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:52.616627 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.616801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:20:52.813330 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.813295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hptqd"]
Sep 18 20:20:52.813514 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.813502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:52.813644 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:52.813623    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:52.817879 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.817522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rhttn"]
Sep 18 20:20:52.817879 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.817647    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:52.817879 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:52.817782    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:52.818178 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.818159    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b8ng8"]
Sep 18 20:20:52.818251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:52.818241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:52.818336 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:52.818321    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:53.599393 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:53.599363    2577 generic.go:356] "Generic (PLEG): container finished" podID="d77a339a-d325-4683-a6ae-945a5b0ebae1" containerID="9a93746472de7e09b41fbdf8672cb49e4fee4d7a7639e8ca5f984c17491b7f72" exitCode=0
Sep 18 20:20:53.599777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:53.599444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" event={"ID":"d77a339a-d325-4683-a6ae-945a5b0ebae1","Type":"ContainerDied","Data":"9a93746472de7e09b41fbdf8672cb49e4fee4d7a7639e8ca5f984c17491b7f72"}
Sep 18 20:20:54.237467 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:54.237437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:54.237642 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:54.237555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:54.237642 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:54.237563    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:54.237642 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:54.237590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:54.237759 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:54.237688    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:54.237799 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:54.237784    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:54.604327 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:54.604289    2577 generic.go:356] "Generic (PLEG): container finished" podID="d77a339a-d325-4683-a6ae-945a5b0ebae1" containerID="fba9761d5bc93be232940fcedbfe49da37f521e5733983a66b995fb0203f40a9" exitCode=0
Sep 18 20:20:54.604820 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:54.604669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" event={"ID":"d77a339a-d325-4683-a6ae-945a5b0ebae1","Type":"ContainerDied","Data":"fba9761d5bc93be232940fcedbfe49da37f521e5733983a66b995fb0203f40a9"}
Sep 18 20:20:55.375249 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:55.375216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:55.475112 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:55.475073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:56.238488 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:56.238453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:56.238973 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:56.238459    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:56.238973 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:56.238464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:56.238973 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:56.238661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:56.238973 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:56.238768    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:56.238973 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:56.238877    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:56.338687 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:56.338650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:56.398489 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:56.398435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:56.837232 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:56.837200    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:56.895004 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:56.894963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:57.767619 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:57.767583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:57.873641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:57.873599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:57.910156 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:57.910114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:57.910335 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:57.910196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:57.910335 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:57.910288    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:57.910442 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:57.910337    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:57.910442 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:57.910369    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs podName:7795f7f8-da44-460a-a884-07d834c38ba3 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:29.910345894 +0000 UTC m=+66.720131816 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs") pod "network-metrics-daemon-rhttn" (UID: "7795f7f8-da44-460a-a884-07d834c38ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:57.910442 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:57.910391    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret podName:27bf6b08-ab66-451f-bc9b-ce188564686c nodeName:}" failed. No retries permitted until 2026-09-18 20:21:29.91038185 +0000 UTC m=+66.720167774 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret") pod "global-pull-secret-syncer-b8ng8" (UID: "27bf6b08-ab66-451f-bc9b-ce188564686c") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:58.011369 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.011328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:58.011574 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.011535    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:58.011574 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.011563    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:58.011574 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.011576    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft7fq for pod openshift-network-diagnostics/network-check-target-hptqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:58.011725 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.011642    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq podName:fa298501-12e5-49d3-9272-b6a6f045d362 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:30.011622072 +0000 UTC m=+66.821408014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft7fq" (UniqueName: "kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq") pod "network-check-target-hptqd" (UID: "fa298501-12e5-49d3-9272-b6a6f045d362") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:58.238552 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.238522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:20:58.238552 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.238545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:20:58.238797 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.238522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:20:58.238797 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.238643    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8ng8" podUID="27bf6b08-ab66-451f-bc9b-ce188564686c"
Sep 18 20:20:58.238797 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.238744    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptqd" podUID="fa298501-12e5-49d3-9272-b6a6f045d362"
Sep 18 20:20:58.238941 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.238841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhttn" podUID="7795f7f8-da44-460a-a884-07d834c38ba3"
Sep 18 20:20:58.638847 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.638821    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeReady"
Sep 18 20:20:58.638996 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.638924    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 20:20:58.675884 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.675858    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54747995bd-wsbr4"]
Sep 18 20:20:58.709583 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.709549    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7xtmj"]
Sep 18 20:20:58.709748 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.709669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.713198 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.713174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 20:20:58.713664 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.713648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 20:20:58.713760 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.713746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wlxzm\""
Sep 18 20:20:58.713990 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.713976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 20:20:58.720876 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.720857    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 20:20:58.723742 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.723724    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l6gq4"]
Sep 18 20:20:58.723836 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.723822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.727856 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.727836    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9622\""
Sep 18 20:20:58.727931 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.727877    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 20:20:58.728435 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.728417    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 20:20:58.754087 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.754068    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54747995bd-wsbr4"]
Sep 18 20:20:58.754181 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.754094    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l6gq4"]
Sep 18 20:20:58.754181 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.754108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7xtmj"]
Sep 18 20:20:58.754257 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.754181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l6gq4"
Sep 18 20:20:58.757810 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.757794    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 20:20:58.758057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.758042    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 20:20:58.758107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.758063    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jdpfx\""
Sep 18 20:20:58.758930 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.758916    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 20:20:58.816384 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e14693-08d8-4f26-bc83-a0ca0dcd8073-tmp-dir\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-certificates\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt6d7\" (UniqueName: \"kubernetes.io/projected/79e14693-08d8-4f26-bc83-a0ca0dcd8073-kube-api-access-jt6d7\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-bound-sa-token\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-trusted-ca\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4glm\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-kube-api-access-j4glm\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-ca-trust-extracted\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-installation-pull-secrets\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-image-registry-private-configuration\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.816709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79e14693-08d8-4f26-bc83-a0ca0dcd8073-config-volume\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.817045 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.816737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.853245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.853213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:58.917814 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.917787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e14693-08d8-4f26-bc83-a0ca0dcd8073-tmp-dir\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.917932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.917830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-certificates\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.917932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.917861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6d7\" (UniqueName: \"kubernetes.io/projected/79e14693-08d8-4f26-bc83-a0ca0dcd8073-kube-api-access-jt6d7\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.917932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.917922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-bound-sa-token\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.918089 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.917945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-trusted-ca\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.918089 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4glm\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-kube-api-access-j4glm\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.918192 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-ca-trust-extracted\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.918192 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c36bc1f-ebd4-4c94-aa48-51ed310e349f-cert\") pod \"ingress-canary-l6gq4\" (UID: \"5c36bc1f-ebd4-4c94-aa48-51ed310e349f\") " pod="openshift-ingress-canary/ingress-canary-l6gq4"
Sep 18 20:20:58.918291 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e14693-08d8-4f26-bc83-a0ca0dcd8073-tmp-dir\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.918291 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-installation-pull-secrets\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.918397 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-image-registry-private-configuration\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.918397 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.918397 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79e14693-08d8-4f26-bc83-a0ca0dcd8073-config-volume\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.918579 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87cxr\" (UniqueName: \"kubernetes.io/projected/5c36bc1f-ebd4-4c94-aa48-51ed310e349f-kube-api-access-87cxr\") pod \"ingress-canary-l6gq4\" (UID: \"5c36bc1f-ebd4-4c94-aa48-51ed310e349f\") " pod="openshift-ingress-canary/ingress-canary-l6gq4"
Sep 18 20:20:58.918579 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.918432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.918685 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.918596    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:20:58.918685 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.918667    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls podName:79e14693-08d8-4f26-bc83-a0ca0dcd8073 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:59.418645907 +0000 UTC m=+36.228431841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls") pod "dns-default-7xtmj" (UID: "79e14693-08d8-4f26-bc83-a0ca0dcd8073") : secret "dns-default-metrics-tls" not found
Sep 18 20:20:58.920230 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.919017    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:20:58.920230 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.919039    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54747995bd-wsbr4: secret "image-registry-tls" not found
Sep 18 20:20:58.920230 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:58.919104    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls podName:8ac5fd59-a194-4364-a4f6-23da3f6a7ec6 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:59.419084494 +0000 UTC m=+36.228870433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls") pod "image-registry-54747995bd-wsbr4" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6") : secret "image-registry-tls" not found
Sep 18 20:20:58.920230 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.919356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/79e14693-08d8-4f26-bc83-a0ca0dcd8073-config-volume\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.930228 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.930192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-ca-trust-extracted\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.930571 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.930543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-image-registry-private-configuration\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.930663 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.930624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6d7\" (UniqueName: \"kubernetes.io/projected/79e14693-08d8-4f26-bc83-a0ca0dcd8073-kube-api-access-jt6d7\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:58.930721 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.930711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4glm\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-kube-api-access-j4glm\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.930929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.930905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-installation-pull-secrets\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.930929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.930920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-trusted-ca\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.931047 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.930977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-bound-sa-token\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.939510 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.939463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-certificates\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:58.950763 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:58.950737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:59.019067 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.018925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87cxr\" (UniqueName: \"kubernetes.io/projected/5c36bc1f-ebd4-4c94-aa48-51ed310e349f-kube-api-access-87cxr\") pod \"ingress-canary-l6gq4\" (UID: \"5c36bc1f-ebd4-4c94-aa48-51ed310e349f\") " pod="openshift-ingress-canary/ingress-canary-l6gq4"
Sep 18 20:20:59.019182 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.019135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c36bc1f-ebd4-4c94-aa48-51ed310e349f-cert\") pod \"ingress-canary-l6gq4\" (UID: \"5c36bc1f-ebd4-4c94-aa48-51ed310e349f\") " pod="openshift-ingress-canary/ingress-canary-l6gq4"
Sep 18 20:20:59.021693 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.021670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c36bc1f-ebd4-4c94-aa48-51ed310e349f-cert\") pod \"ingress-canary-l6gq4\" (UID: \"5c36bc1f-ebd4-4c94-aa48-51ed310e349f\") " pod="openshift-ingress-canary/ingress-canary-l6gq4"
Sep 18 20:20:59.029715 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.029696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87cxr\" (UniqueName: \"kubernetes.io/projected/5c36bc1f-ebd4-4c94-aa48-51ed310e349f-kube-api-access-87cxr\") pod \"ingress-canary-l6gq4\" (UID: \"5c36bc1f-ebd4-4c94-aa48-51ed310e349f\") " pod="openshift-ingress-canary/ingress-canary-l6gq4"
Sep 18 20:20:59.061826 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.061795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l6gq4"
Sep 18 20:20:59.389123 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.389092    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l6gq4"]
Sep 18 20:20:59.392367 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:20:59.392332    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c36bc1f_ebd4_4c94_aa48_51ed310e349f.slice/crio-d4d67bc462dd5334f0d0352f0d4d0711f556fd41689e6025cb49e4ba4896943a WatchSource:0}: Error finding container d4d67bc462dd5334f0d0352f0d4d0711f556fd41689e6025cb49e4ba4896943a: Status 404 returned error can't find the container with id d4d67bc462dd5334f0d0352f0d4d0711f556fd41689e6025cb49e4ba4896943a
Sep 18 20:20:59.421299 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.421279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:20:59.421362 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.421320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:20:59.421426 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:59.421416    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:20:59.421464 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:59.421429    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:20:59.421464 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:59.421447    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54747995bd-wsbr4: secret "image-registry-tls" not found
Sep 18 20:20:59.421549 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:59.421469    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls podName:79e14693-08d8-4f26-bc83-a0ca0dcd8073 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:00.421456556 +0000 UTC m=+37.231242475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls") pod "dns-default-7xtmj" (UID: "79e14693-08d8-4f26-bc83-a0ca0dcd8073") : secret "dns-default-metrics-tls" not found
Sep 18 20:20:59.421588 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:20:59.421562    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls podName:8ac5fd59-a194-4364-a4f6-23da3f6a7ec6 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:00.421542309 +0000 UTC m=+37.231328229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls") pod "image-registry-54747995bd-wsbr4" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6") : secret "image-registry-tls" not found
Sep 18 20:20:59.616053 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.616018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l6gq4" event={"ID":"5c36bc1f-ebd4-4c94-aa48-51ed310e349f","Type":"ContainerStarted","Data":"d4d67bc462dd5334f0d0352f0d4d0711f556fd41689e6025cb49e4ba4896943a"}
Sep 18 20:20:59.618369 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.618341    2577 generic.go:356] "Generic (PLEG): container finished" podID="d77a339a-d325-4683-a6ae-945a5b0ebae1" containerID="b8687b03102bcbd3dbdcebf089bd0493bf967ffe1d7070db7e697c314ac51104" exitCode=0
Sep 18 20:20:59.618489 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:20:59.618371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" event={"ID":"d77a339a-d325-4683-a6ae-945a5b0ebae1","Type":"ContainerDied","Data":"b8687b03102bcbd3dbdcebf089bd0493bf967ffe1d7070db7e697c314ac51104"}
Sep 18 20:21:00.238121 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.238094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:21:00.238974 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.238094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:21:00.238974 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.238094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:21:00.241671 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.241648    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 20:21:00.243143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.242848    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 20:21:00.243143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.242857    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 20:21:00.243143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.242866    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 20:21:00.243143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.242854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnbxc\""
Sep 18 20:21:00.243143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.242896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r5tl8\""
Sep 18 20:21:00.429647 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.429616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:00.429811 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.429677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:00.429862 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:00.429816    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:00.429862 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:00.429842    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:00.429936 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:00.429865    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54747995bd-wsbr4: secret "image-registry-tls" not found
Sep 18 20:21:00.429936 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:00.429892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls podName:79e14693-08d8-4f26-bc83-a0ca0dcd8073 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:02.429870299 +0000 UTC m=+39.239656244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls") pod "dns-default-7xtmj" (UID: "79e14693-08d8-4f26-bc83-a0ca0dcd8073") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:00.429936 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:00.429918    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls podName:8ac5fd59-a194-4364-a4f6-23da3f6a7ec6 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:02.42990106 +0000 UTC m=+39.239686984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls") pod "image-registry-54747995bd-wsbr4" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6") : secret "image-registry-tls" not found
Sep 18 20:21:00.624391 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.624353    2577 generic.go:356] "Generic (PLEG): container finished" podID="d77a339a-d325-4683-a6ae-945a5b0ebae1" containerID="8afbf73b7f199a00c9def69782c87b85de378ce5453978004b5c331a23df3ebf" exitCode=0
Sep 18 20:21:00.624600 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:00.624437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" event={"ID":"d77a339a-d325-4683-a6ae-945a5b0ebae1","Type":"ContainerDied","Data":"8afbf73b7f199a00c9def69782c87b85de378ce5453978004b5c331a23df3ebf"}
Sep 18 20:21:01.630383 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:01.630349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" event={"ID":"d77a339a-d325-4683-a6ae-945a5b0ebae1","Type":"ContainerStarted","Data":"01862c837cba2909a476db2a3e03b4404e1b96642db83c86bead3be2a0bf12f1"}
Sep 18 20:21:01.658527 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:01.658455    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v2nh2" podStartSLOduration=7.74865923 podStartE2EDuration="36.65844053s" totalImagesPullingTime="28.9097813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:01.657750332 +0000 UTC m=+38.467536275" watchObservedRunningTime="2026-09-18 20:21:01.65844053 +0000 UTC m=+38.468226476"
Sep 18 20:21:02.446117 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:02.446081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:02.446278 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:02.446192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:02.446278 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:02.446265    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:02.446383 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:02.446296    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:02.446383 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:02.446312    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54747995bd-wsbr4: secret "image-registry-tls" not found
Sep 18 20:21:02.446383 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:02.446337    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls podName:79e14693-08d8-4f26-bc83-a0ca0dcd8073 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:06.446317291 +0000 UTC m=+43.256103228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls") pod "dns-default-7xtmj" (UID: "79e14693-08d8-4f26-bc83-a0ca0dcd8073") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:02.446383 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:02.446365    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls podName:8ac5fd59-a194-4364-a4f6-23da3f6a7ec6 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:06.44634704 +0000 UTC m=+43.256132962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls") pod "image-registry-54747995bd-wsbr4" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6") : secret "image-registry-tls" not found
Sep 18 20:21:02.634337 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:02.634295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l6gq4" event={"ID":"5c36bc1f-ebd4-4c94-aa48-51ed310e349f","Type":"ContainerStarted","Data":"a5e0e64b4df6cb73ee103797e13eb04c082f9b6b6c4039056c3d5209844ea039"}
Sep 18 20:21:02.652311 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:02.652214    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l6gq4" podStartSLOduration=2.344168636 podStartE2EDuration="4.652183403s" totalImagesPullingTime="2.308014767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:02.651262139 +0000 UTC m=+39.461048080" watchObservedRunningTime="2026-09-18 20:21:02.652183403 +0000 UTC m=+39.461969347"
Sep 18 20:21:06.476184 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:06.476142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:06.476669 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:06.476194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:06.476669 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:06.476293    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:06.476669 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:06.476315    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:06.476669 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:06.476340    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54747995bd-wsbr4: secret "image-registry-tls" not found
Sep 18 20:21:06.476669 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:06.476346    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls podName:79e14693-08d8-4f26-bc83-a0ca0dcd8073 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:14.476330344 +0000 UTC m=+51.286116263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls") pod "dns-default-7xtmj" (UID: "79e14693-08d8-4f26-bc83-a0ca0dcd8073") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:06.476669 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:06.476413    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls podName:8ac5fd59-a194-4364-a4f6-23da3f6a7ec6 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:14.476393884 +0000 UTC m=+51.286179805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls") pod "image-registry-54747995bd-wsbr4" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6") : secret "image-registry-tls" not found
Sep 18 20:21:07.821549 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.821512    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ltk7w"]
Sep 18 20:21:07.843848 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.843811    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ltk7w"]
Sep 18 20:21:07.844008 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.843908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:07.846325 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.846301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 20:21:07.846866 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.846842    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 20:21:07.846981 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.846964    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 20:21:07.847388 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.847370    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 20:21:07.848372 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.848355    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4bqd\""
Sep 18 20:21:07.987288 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.987256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wms4p\" (UniqueName: \"kubernetes.io/projected/385a1e4e-2987-4cfd-acf7-69508486b476-kube-api-access-wms4p\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:07.987288 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.987294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:07.987581 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.987327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/385a1e4e-2987-4cfd-acf7-69508486b476-crio-socket\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:07.987581 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.987458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:07.987581 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:07.987554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/385a1e4e-2987-4cfd-acf7-69508486b476-data-volume\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.088393 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.088363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.088585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.088410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/385a1e4e-2987-4cfd-acf7-69508486b476-data-volume\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.088585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.088448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wms4p\" (UniqueName: \"kubernetes.io/projected/385a1e4e-2987-4cfd-acf7-69508486b476-kube-api-access-wms4p\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.088585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.088467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.088585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.088505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/385a1e4e-2987-4cfd-acf7-69508486b476-crio-socket\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.088585 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:08.088556    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:08.088809 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:08.088599    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:08.088809 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:08.088628    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:08.588607476 +0000 UTC m=+45.398393396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:08.088809 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:08.088650    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:08.588637574 +0000 UTC m=+45.398423494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:08.088809 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.088743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/385a1e4e-2987-4cfd-acf7-69508486b476-crio-socket\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.088933 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.088826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/385a1e4e-2987-4cfd-acf7-69508486b476-data-volume\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.099242 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.099216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wms4p\" (UniqueName: \"kubernetes.io/projected/385a1e4e-2987-4cfd-acf7-69508486b476-kube-api-access-wms4p\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.591769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.591736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.592002 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:08.591802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:08.592002 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:08.591862    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:08.592002 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:08.591893    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:08.592002 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:08.591935    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:09.591916808 +0000 UTC m=+46.401702745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:08.592002 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:08.591955    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:09.591941847 +0000 UTC m=+46.401727767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:09.032666 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.032595    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc"]
Sep 18 20:21:09.057373 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.057350    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc"]
Sep 18 20:21:09.057504 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.057442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc"
Sep 18 20:21:09.060037 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.060016    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rt84g\""
Sep 18 20:21:09.061047 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.061026    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 20:21:09.061159 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.061046    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 20:21:09.167857 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.167837    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kpgfw_244a261e-ab8c-41a4-9a04-09b22da8a488/dns-node-resolver/0.log"
Sep 18 20:21:09.195088 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.195063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb675\" (UniqueName: \"kubernetes.io/projected/a0e95428-8170-4f7e-b84c-26bcf40dbd69-kube-api-access-nb675\") pod \"migrator-76794cbd6c-wwknc\" (UID: \"a0e95428-8170-4f7e-b84c-26bcf40dbd69\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc"
Sep 18 20:21:09.296206 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.296136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb675\" (UniqueName: \"kubernetes.io/projected/a0e95428-8170-4f7e-b84c-26bcf40dbd69-kube-api-access-nb675\") pod \"migrator-76794cbd6c-wwknc\" (UID: \"a0e95428-8170-4f7e-b84c-26bcf40dbd69\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc"
Sep 18 20:21:09.304374 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.304348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb675\" (UniqueName: \"kubernetes.io/projected/a0e95428-8170-4f7e-b84c-26bcf40dbd69-kube-api-access-nb675\") pod \"migrator-76794cbd6c-wwknc\" (UID: \"a0e95428-8170-4f7e-b84c-26bcf40dbd69\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc"
Sep 18 20:21:09.366466 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.366439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc"
Sep 18 20:21:09.598986 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.598959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:09.599139 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.599021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:09.599139 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:09.599095    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:09.599213 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:09.599166    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:11.599147441 +0000 UTC m=+48.408933382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:09.599213 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:09.599166    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:09.599213 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:09.599204    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:11.599193845 +0000 UTC m=+48.408979765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:09.667747 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:09.667715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc"]
Sep 18 20:21:09.670235 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:09.670197    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e95428_8170_4f7e_b84c_26bcf40dbd69.slice/crio-bcf928db6dec07aba8263187cecebfd652d08514c3270be7df562de28deb7bb1 WatchSource:0}: Error finding container bcf928db6dec07aba8263187cecebfd652d08514c3270be7df562de28deb7bb1: Status 404 returned error can't find the container with id bcf928db6dec07aba8263187cecebfd652d08514c3270be7df562de28deb7bb1
Sep 18 20:21:10.167512 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.167467    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gzrh4_7fa951d2-1593-4871-b3b3-76614e660378/node-ca/0.log"
Sep 18 20:21:10.458603 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.458522    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xjlqh"]
Sep 18 20:21:10.473772 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.473749    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xjlqh"]
Sep 18 20:21:10.473924 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.473849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.476548 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.476527    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 20:21:10.477617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.477406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 20:21:10.477617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.477406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 20:21:10.477617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.477450    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 20:21:10.477617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.477412    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lkqbp\""
Sep 18 20:21:10.606322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.606295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29a50244-3ca5-45d8-ae0e-3b37597d4c78-signing-key\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.606506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.606370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29a50244-3ca5-45d8-ae0e-3b37597d4c78-signing-cabundle\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.606506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.606457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf77r\" (UniqueName: \"kubernetes.io/projected/29a50244-3ca5-45d8-ae0e-3b37597d4c78-kube-api-access-kf77r\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.654449 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.654416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc" event={"ID":"a0e95428-8170-4f7e-b84c-26bcf40dbd69","Type":"ContainerStarted","Data":"bcf928db6dec07aba8263187cecebfd652d08514c3270be7df562de28deb7bb1"}
Sep 18 20:21:10.707847 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.707802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29a50244-3ca5-45d8-ae0e-3b37597d4c78-signing-key\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.708012 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.707949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29a50244-3ca5-45d8-ae0e-3b37597d4c78-signing-cabundle\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.708012 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.707992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf77r\" (UniqueName: \"kubernetes.io/projected/29a50244-3ca5-45d8-ae0e-3b37597d4c78-kube-api-access-kf77r\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.708581 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.708560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29a50244-3ca5-45d8-ae0e-3b37597d4c78-signing-cabundle\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.710270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.710215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29a50244-3ca5-45d8-ae0e-3b37597d4c78-signing-key\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.716399 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.716381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf77r\" (UniqueName: \"kubernetes.io/projected/29a50244-3ca5-45d8-ae0e-3b37597d4c78-kube-api-access-kf77r\") pod \"service-ca-55f6fc7b86-xjlqh\" (UID: \"29a50244-3ca5-45d8-ae0e-3b37597d4c78\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.784977 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.784937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh"
Sep 18 20:21:10.967878 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:10.967815    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l6gq4_5c36bc1f-ebd4-4c94-aa48-51ed310e349f/serve-healthcheck-canary/0.log"
Sep 18 20:21:11.314533 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:11.314456    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xjlqh"]
Sep 18 20:21:11.420260 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:11.420232    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a50244_3ca5_45d8_ae0e_3b37597d4c78.slice/crio-9fda1078def5a539fb772f4485cdfd082491050607be014543a366d2fc7a33f5 WatchSource:0}: Error finding container 9fda1078def5a539fb772f4485cdfd082491050607be014543a366d2fc7a33f5: Status 404 returned error can't find the container with id 9fda1078def5a539fb772f4485cdfd082491050607be014543a366d2fc7a33f5
Sep 18 20:21:11.616315 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:11.616286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:11.616417 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:11.616339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:11.616512 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:11.616411    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:11.616512 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:11.616431    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:11.616512 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:11.616469    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:15.616452795 +0000 UTC m=+52.426238736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:11.616512 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:11.616508    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:15.616499655 +0000 UTC m=+52.426285575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:11.658048 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:11.658027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh" event={"ID":"29a50244-3ca5-45d8-ae0e-3b37597d4c78","Type":"ContainerStarted","Data":"9fda1078def5a539fb772f4485cdfd082491050607be014543a366d2fc7a33f5"}
Sep 18 20:21:11.659750 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:11.659731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc" event={"ID":"a0e95428-8170-4f7e-b84c-26bcf40dbd69","Type":"ContainerStarted","Data":"91ccb0a7b94054aefd5dd77a90caf7961a701014c5dceb4254fa571f8835fc57"}
Sep 18 20:21:11.659826 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:11.659768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc" event={"ID":"a0e95428-8170-4f7e-b84c-26bcf40dbd69","Type":"ContainerStarted","Data":"2a6e5805b39f60c039027ec111c03a4037981394038b2352fa2b10666d611278"}
Sep 18 20:21:12.687661 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:12.687608    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wwknc" podStartSLOduration=1.908814896 podStartE2EDuration="3.687592588s" totalImagesPullingTime="1.778777692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:12.687297609 +0000 UTC m=+49.497083550" watchObservedRunningTime="2026-09-18 20:21:12.687592588 +0000 UTC m=+49.497378530"
Sep 18 20:21:14.540903 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:14.540864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:14.541251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:14.540914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:14.541251 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:14.541033    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:14.541251 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:14.541080    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls podName:79e14693-08d8-4f26-bc83-a0ca0dcd8073 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:30.541067363 +0000 UTC m=+67.350853282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls") pod "dns-default-7xtmj" (UID: "79e14693-08d8-4f26-bc83-a0ca0dcd8073") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:14.541251 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:14.541029    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:14.541251 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:14.541130    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54747995bd-wsbr4: secret "image-registry-tls" not found
Sep 18 20:21:14.541251 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:14.541185    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls podName:8ac5fd59-a194-4364-a4f6-23da3f6a7ec6 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:30.541170498 +0000 UTC m=+67.350956418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls") pod "image-registry-54747995bd-wsbr4" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6") : secret "image-registry-tls" not found
Sep 18 20:21:14.668730 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:14.668696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh" event={"ID":"29a50244-3ca5-45d8-ae0e-3b37597d4c78","Type":"ContainerStarted","Data":"18535ac166ae6eccf1a5068a4d6e9f0dd3ce1b893bb3fc3c0ec8afc04b48ebbd"}
Sep 18 20:21:14.685824 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:14.685780    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-xjlqh" podStartSLOduration=2.247766312 podStartE2EDuration="4.685765572s" totalImagesPullingTime="2.43799926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:14.684686092 +0000 UTC m=+51.494472033" watchObservedRunningTime="2026-09-18 20:21:14.685765572 +0000 UTC m=+51.495551514"
Sep 18 20:21:15.650793 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:15.650759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:15.651167 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:15.650825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:15.651167 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:15.650895    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:15.651167 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:15.650970    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:23.650955378 +0000 UTC m=+60.460741304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:15.651167 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:15.651012    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:15.651167 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:15.651085    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:23.651063177 +0000 UTC m=+60.460849102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:23.709595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:23.709553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:23.709995 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:23.709621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:23.709995 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:23.709715    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:23.709995 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:21:23.709766    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls podName:385a1e4e-2987-4cfd-acf7-69508486b476 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:39.709751412 +0000 UTC m=+76.519537332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltk7w" (UID: "385a1e4e-2987-4cfd-acf7-69508486b476") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:23.710101 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:23.710085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/385a1e4e-2987-4cfd-acf7-69508486b476-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:24.615041 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:24.615015    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dklfc"
Sep 18 20:21:29.954335 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:29.954295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:21:29.954750 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:29.954353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:21:29.957033 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:29.957012    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 20:21:29.957090 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:29.957067    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 20:21:29.967114 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:29.967084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27bf6b08-ab66-451f-bc9b-ce188564686c-original-pull-secret\") pod \"global-pull-secret-syncer-b8ng8\" (UID: \"27bf6b08-ab66-451f-bc9b-ce188564686c\") " pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:21:29.967219 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:29.967116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7795f7f8-da44-460a-a884-07d834c38ba3-metrics-certs\") pod \"network-metrics-daemon-rhttn\" (UID: \"7795f7f8-da44-460a-a884-07d834c38ba3\") " pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:21:30.055731 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.055683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:21:30.058106 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.058089    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 20:21:30.068783 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.068766    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 20:21:30.079033 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.079006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7fq\" (UniqueName: \"kubernetes.io/projected/fa298501-12e5-49d3-9272-b6a6f045d362-kube-api-access-ft7fq\") pod \"network-check-target-hptqd\" (UID: \"fa298501-12e5-49d3-9272-b6a6f045d362\") " pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:21:30.252361 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.252290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnbxc\""
Sep 18 20:21:30.258581 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.258564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r5tl8\""
Sep 18 20:21:30.260346 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.260317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhttn"
Sep 18 20:21:30.262570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.262548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8ng8"
Sep 18 20:21:30.266447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.266434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:21:30.533206 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.533123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b8ng8"]
Sep 18 20:21:30.536122 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:30.536090    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bf6b08_ab66_451f_bc9b_ce188564686c.slice/crio-517906ec11890635460c1fd474bc5e481209b4c88b84211f80a2d27356d55999 WatchSource:0}: Error finding container 517906ec11890635460c1fd474bc5e481209b4c88b84211f80a2d27356d55999: Status 404 returned error can't find the container with id 517906ec11890635460c1fd474bc5e481209b4c88b84211f80a2d27356d55999
Sep 18 20:21:30.537007 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.536983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rhttn"]
Sep 18 20:21:30.539336 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.539248    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hptqd"]
Sep 18 20:21:30.539763 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:30.539729    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7795f7f8_da44_460a_a884_07d834c38ba3.slice/crio-43048dfdfa7a38a76400257d41205fa16f1f26645929258eb557f95c51358cf8 WatchSource:0}: Error finding container 43048dfdfa7a38a76400257d41205fa16f1f26645929258eb557f95c51358cf8: Status 404 returned error can't find the container with id 43048dfdfa7a38a76400257d41205fa16f1f26645929258eb557f95c51358cf8
Sep 18 20:21:30.543990 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:30.543969    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa298501_12e5_49d3_9272_b6a6f045d362.slice/crio-08dea75e0fd19b6ed3d9bc16d19b13f4475c6e38192cbf920b027450b1b6fada WatchSource:0}: Error finding container 08dea75e0fd19b6ed3d9bc16d19b13f4475c6e38192cbf920b027450b1b6fada: Status 404 returned error can't find the container with id 08dea75e0fd19b6ed3d9bc16d19b13f4475c6e38192cbf920b027450b1b6fada
Sep 18 20:21:30.558611 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.558588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:30.558703 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.558625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:30.560802 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.560778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/79e14693-08d8-4f26-bc83-a0ca0dcd8073-metrics-tls\") pod \"dns-default-7xtmj\" (UID: \"79e14693-08d8-4f26-bc83-a0ca0dcd8073\") " pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:30.560896 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.560877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"image-registry-54747995bd-wsbr4\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") " pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:30.706677 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.706645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hptqd" event={"ID":"fa298501-12e5-49d3-9272-b6a6f045d362","Type":"ContainerStarted","Data":"08dea75e0fd19b6ed3d9bc16d19b13f4475c6e38192cbf920b027450b1b6fada"}
Sep 18 20:21:30.707774 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.707745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b8ng8" event={"ID":"27bf6b08-ab66-451f-bc9b-ce188564686c","Type":"ContainerStarted","Data":"517906ec11890635460c1fd474bc5e481209b4c88b84211f80a2d27356d55999"}
Sep 18 20:21:30.708884 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.708864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhttn" event={"ID":"7795f7f8-da44-460a-a884-07d834c38ba3","Type":"ContainerStarted","Data":"43048dfdfa7a38a76400257d41205fa16f1f26645929258eb557f95c51358cf8"}
Sep 18 20:21:30.821844 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.821761    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wlxzm\""
Sep 18 20:21:30.830406 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.830376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:30.833654 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.833637    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9622\""
Sep 18 20:21:30.842575 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:30.842547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:31.111193 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:31.111117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54747995bd-wsbr4"]
Sep 18 20:21:31.115266 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:31.115229    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac5fd59_a194_4364_a4f6_23da3f6a7ec6.slice/crio-1c39e4055429318f41bca3ab2b2f15797e63270eee7f79d946bd2f6a1c573853 WatchSource:0}: Error finding container 1c39e4055429318f41bca3ab2b2f15797e63270eee7f79d946bd2f6a1c573853: Status 404 returned error can't find the container with id 1c39e4055429318f41bca3ab2b2f15797e63270eee7f79d946bd2f6a1c573853
Sep 18 20:21:31.131227 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:31.131180    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7xtmj"]
Sep 18 20:21:31.137749 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:31.137683    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e14693_08d8_4f26_bc83_a0ca0dcd8073.slice/crio-2dc26f34cd050714a9296987ce8076c0d1115b5b1e5ff154e49417f917e2a79d WatchSource:0}: Error finding container 2dc26f34cd050714a9296987ce8076c0d1115b5b1e5ff154e49417f917e2a79d: Status 404 returned error can't find the container with id 2dc26f34cd050714a9296987ce8076c0d1115b5b1e5ff154e49417f917e2a79d
Sep 18 20:21:31.715642 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:31.715605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7xtmj" event={"ID":"79e14693-08d8-4f26-bc83-a0ca0dcd8073","Type":"ContainerStarted","Data":"2dc26f34cd050714a9296987ce8076c0d1115b5b1e5ff154e49417f917e2a79d"}
Sep 18 20:21:31.718143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:31.717578    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54747995bd-wsbr4" event={"ID":"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6","Type":"ContainerStarted","Data":"fee39ba142083135cf7368a46fa4af62cf43b1672a221fe76974c205414ed4ea"}
Sep 18 20:21:31.718143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:31.717631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54747995bd-wsbr4" event={"ID":"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6","Type":"ContainerStarted","Data":"1c39e4055429318f41bca3ab2b2f15797e63270eee7f79d946bd2f6a1c573853"}
Sep 18 20:21:31.718143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:31.717797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:32.724413 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:32.723908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhttn" event={"ID":"7795f7f8-da44-460a-a884-07d834c38ba3","Type":"ContainerStarted","Data":"6bf552a2bb1065bc3cefecd1f7613a375902846fc664e82831d40ce6de49a405"}
Sep 18 20:21:32.724413 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:32.724360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhttn" event={"ID":"7795f7f8-da44-460a-a884-07d834c38ba3","Type":"ContainerStarted","Data":"513eb709ad7ad5bf08211519ba1bf8b419c5027f4ec4423d7116b377239977e0"}
Sep 18 20:21:32.793208 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:32.793147    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54747995bd-wsbr4" podStartSLOduration=71.793121777 podStartE2EDuration="1m11.793121777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:31.744460111 +0000 UTC m=+68.554246054" watchObservedRunningTime="2026-09-18 20:21:32.793121777 +0000 UTC m=+69.602907721"
Sep 18 20:21:32.915017 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:32.914947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rhttn" podStartSLOduration=66.818456255 podStartE2EDuration="1m7.914922946s" totalImagesPullingTime="1.096466691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:32.794211447 +0000 UTC m=+69.603997389" watchObservedRunningTime="2026-09-18 20:21:32.914922946 +0000 UTC m=+69.724708888"
Sep 18 20:21:32.915883 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:32.915859    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54747995bd-wsbr4"]
Sep 18 20:21:35.736332 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.736291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7xtmj" event={"ID":"79e14693-08d8-4f26-bc83-a0ca0dcd8073","Type":"ContainerStarted","Data":"3597c974e88295e41c535415a82766ae55078510153e7cf673abab511c2c9125"}
Sep 18 20:21:35.736830 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.736388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7xtmj" event={"ID":"79e14693-08d8-4f26-bc83-a0ca0dcd8073","Type":"ContainerStarted","Data":"68e452384efb00b9cff3e9a8aa0b9fbcbaf7ab4c2e26e3e9bc20ef9e59d283a7"}
Sep 18 20:21:35.736954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.736934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:35.739644 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.739617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hptqd" event={"ID":"fa298501-12e5-49d3-9272-b6a6f045d362","Type":"ContainerStarted","Data":"d0eff881cb9bebef221fcd3c87ac7087029e9dabe9aff9b1b87842653f3c9571"}
Sep 18 20:21:35.739749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.739677    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:21:35.741221 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.741189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b8ng8" event={"ID":"27bf6b08-ab66-451f-bc9b-ce188564686c","Type":"ContainerStarted","Data":"372bfb2eec2884b2d2e1d9aefb8464a2b88d59efca9dbf2e39699a8dd5aa9e34"}
Sep 18 20:21:35.799963 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.799853    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b8ng8" podStartSLOduration=65.792992934 podStartE2EDuration="1m10.799832257s" totalImagesPullingTime="5.006839323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:35.795586553 +0000 UTC m=+72.605372496" watchObservedRunningTime="2026-09-18 20:21:35.799832257 +0000 UTC m=+72.609618200"
Sep 18 20:21:35.800113 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.800042    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7xtmj" podStartSLOduration=33.411803434 podStartE2EDuration="37.800032547s" totalImagesPullingTime="4.388229113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:35.770840446 +0000 UTC m=+72.580626391" watchObservedRunningTime="2026-09-18 20:21:35.800032547 +0000 UTC m=+72.609818491"
Sep 18 20:21:35.820746 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.820683    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hptqd" podStartSLOduration=65.832664301 podStartE2EDuration="1m10.82066089s" totalImagesPullingTime="4.987996589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:35.81980087 +0000 UTC m=+72.629586813" watchObservedRunningTime="2026-09-18 20:21:35.82066089 +0000 UTC m=+72.630446834"
Sep 18 20:21:35.891621 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.891585    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-559cdf5c69-75fdj"]
Sep 18 20:21:35.894510 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.894469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:35.897204 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.897185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bf7wk\""
Sep 18 20:21:35.897371 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.897354    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 20:21:35.897674 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.897659    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 20:21:35.897931 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.897899    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 20:21:35.898062 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.898044    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 20:21:35.898128 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.898086    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 20:21:35.898211 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.898191    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 20:21:35.898273 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.898195    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 20:21:35.913997 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:35.913975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559cdf5c69-75fdj"]
Sep 18 20:21:36.006377 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.006347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-config\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.006555 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.006387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-service-ca\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.006555 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.006403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-oauth-serving-cert\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.006555 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.006509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-oauth-config\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.006555 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.006534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsh6t\" (UniqueName: \"kubernetes.io/projected/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-kube-api-access-hsh6t\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.006708 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.006567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-serving-cert\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.107626 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.107576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-config\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.107826 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.107638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-service-ca\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.107826 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.107669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-oauth-serving-cert\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.107826 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.107696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-oauth-config\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.107826 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.107718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsh6t\" (UniqueName: \"kubernetes.io/projected/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-kube-api-access-hsh6t\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.107826 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.107747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-serving-cert\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.108373 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.108346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-config\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.108514 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.108432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-service-ca\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.108514 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.108432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-oauth-serving-cert\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.110134 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.110099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-oauth-config\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.110134 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.110131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-serving-cert\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.117022 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.117001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsh6t\" (UniqueName: \"kubernetes.io/projected/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-kube-api-access-hsh6t\") pod \"console-559cdf5c69-75fdj\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") " pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.202811 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.202774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:36.470773 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.470700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559cdf5c69-75fdj"]
Sep 18 20:21:36.473658 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:36.473632    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07762b6_9c63_4cb4_a3c3_38d94dc004c6.slice/crio-0589702caf9a9f8d3e7971c6eb880bfd0898ca1fb46923edd8345bf1d25d4a09 WatchSource:0}: Error finding container 0589702caf9a9f8d3e7971c6eb880bfd0898ca1fb46923edd8345bf1d25d4a09: Status 404 returned error can't find the container with id 0589702caf9a9f8d3e7971c6eb880bfd0898ca1fb46923edd8345bf1d25d4a09
Sep 18 20:21:36.745911 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:36.745817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559cdf5c69-75fdj" event={"ID":"b07762b6-9c63-4cb4-a3c3-38d94dc004c6","Type":"ContainerStarted","Data":"0589702caf9a9f8d3e7971c6eb880bfd0898ca1fb46923edd8345bf1d25d4a09"}
Sep 18 20:21:39.737994 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:39.737955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:39.740537 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:39.740507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/385a1e4e-2987-4cfd-acf7-69508486b476-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltk7w\" (UID: \"385a1e4e-2987-4cfd-acf7-69508486b476\") " pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:39.956546 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:39.956515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4bqd\""
Sep 18 20:21:39.963664 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:39.963636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltk7w"
Sep 18 20:21:40.482507 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:40.482346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ltk7w"]
Sep 18 20:21:40.485139 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:40.485113    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385a1e4e_2987_4cfd_acf7_69508486b476.slice/crio-82525c57c9417fa864cd0f0caf4eebd6f075e7d97ea44e9a9803e3c7b9e17e3e WatchSource:0}: Error finding container 82525c57c9417fa864cd0f0caf4eebd6f075e7d97ea44e9a9803e3c7b9e17e3e: Status 404 returned error can't find the container with id 82525c57c9417fa864cd0f0caf4eebd6f075e7d97ea44e9a9803e3c7b9e17e3e
Sep 18 20:21:40.763676 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:40.763599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltk7w" event={"ID":"385a1e4e-2987-4cfd-acf7-69508486b476","Type":"ContainerStarted","Data":"5bc500c8c9973001d89b7b3d7feca14435d7fa4caae476b9a4ea5e7932e3645a"}
Sep 18 20:21:40.763676 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:40.763663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltk7w" event={"ID":"385a1e4e-2987-4cfd-acf7-69508486b476","Type":"ContainerStarted","Data":"82525c57c9417fa864cd0f0caf4eebd6f075e7d97ea44e9a9803e3c7b9e17e3e"}
Sep 18 20:21:40.765204 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:40.765180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559cdf5c69-75fdj" event={"ID":"b07762b6-9c63-4cb4-a3c3-38d94dc004c6","Type":"ContainerStarted","Data":"8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7"}
Sep 18 20:21:40.809934 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:40.809887    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-559cdf5c69-75fdj" podStartSLOduration=2.007761118 podStartE2EDuration="5.8098736s" totalImagesPullingTime="3.802112482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:40.807831366 +0000 UTC m=+77.617617320" watchObservedRunningTime="2026-09-18 20:21:40.8098736 +0000 UTC m=+77.619659541"
Sep 18 20:21:41.771813 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:41.771732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltk7w" event={"ID":"385a1e4e-2987-4cfd-acf7-69508486b476","Type":"ContainerStarted","Data":"7701a35f7d14b6025ff47d6d6a9078da7f995a8941efe3e2338f584d08464414"}
Sep 18 20:21:42.226494 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.226063    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h8hp2"]
Sep 18 20:21:42.230514 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.230462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.233777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.233748    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 20:21:42.234758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.234610    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fg9vd\""
Sep 18 20:21:42.234758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.234625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 20:21:42.234758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.234640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 20:21:42.234959 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.234899    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 20:21:42.234959 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.234952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 20:21:42.235081 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.234952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 20:21:42.360532 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-sys\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.360718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-tls\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.360718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-wtmp\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.360718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eedd0741-cae1-4c0f-8bb6-616b92088ca6-metrics-client-ca\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.360718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zx6\" (UniqueName: \"kubernetes.io/projected/eedd0741-cae1-4c0f-8bb6-616b92088ca6-kube-api-access-v8zx6\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.360718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.360971 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.360971 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-root\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.360971 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.360815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-textfile\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461396 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-wtmp\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461531 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eedd0741-cae1-4c0f-8bb6-616b92088ca6-metrics-client-ca\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461531 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zx6\" (UniqueName: \"kubernetes.io/projected/eedd0741-cae1-4c0f-8bb6-616b92088ca6-kube-api-access-v8zx6\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461531 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461531 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461531 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-root\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-wtmp\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-textfile\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-sys\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-tls\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.461968 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.461832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-textfile\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.462054 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.462030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eedd0741-cae1-4c0f-8bb6-616b92088ca6-metrics-client-ca\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.462498 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.462317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-root\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.462498 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.462366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.462498 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.462381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eedd0741-cae1-4c0f-8bb6-616b92088ca6-sys\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.464329 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.464309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-tls\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.464430 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.464346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eedd0741-cae1-4c0f-8bb6-616b92088ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.469292 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.469271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zx6\" (UniqueName: \"kubernetes.io/projected/eedd0741-cae1-4c0f-8bb6-616b92088ca6-kube-api-access-v8zx6\") pod \"node-exporter-h8hp2\" (UID: \"eedd0741-cae1-4c0f-8bb6-616b92088ca6\") " pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.542872 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.542835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h8hp2"
Sep 18 20:21:42.553703 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:42.553669    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedd0741_cae1_4c0f_8bb6_616b92088ca6.slice/crio-875e5bad9502570e6d18a38c9ca461a3929e95938aa55458cbe8585715a7df26 WatchSource:0}: Error finding container 875e5bad9502570e6d18a38c9ca461a3929e95938aa55458cbe8585715a7df26: Status 404 returned error can't find the container with id 875e5bad9502570e6d18a38c9ca461a3929e95938aa55458cbe8585715a7df26
Sep 18 20:21:42.776956 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.776872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltk7w" event={"ID":"385a1e4e-2987-4cfd-acf7-69508486b476","Type":"ContainerStarted","Data":"4b09a78fe071531bda0c556c202f9ee3ad5fa393c4f1f5158800e85eef4acadf"}
Sep 18 20:21:42.781035 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.780990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h8hp2" event={"ID":"eedd0741-cae1-4c0f-8bb6-616b92088ca6","Type":"ContainerStarted","Data":"875e5bad9502570e6d18a38c9ca461a3929e95938aa55458cbe8585715a7df26"}
Sep 18 20:21:42.797427 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:42.797372    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ltk7w" podStartSLOduration=33.96029305 podStartE2EDuration="35.797353643s" totalImagesPullingTime="1.837060593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:21:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:42.796206078 +0000 UTC m=+79.605992020" watchObservedRunningTime="2026-09-18 20:21:42.797353643 +0000 UTC m=+79.607139586"
Sep 18 20:21:43.786040 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:43.785963    2577 generic.go:356] "Generic (PLEG): container finished" podID="eedd0741-cae1-4c0f-8bb6-616b92088ca6" containerID="e9bdf7f83a8d0e41cff9d5d6bc779e8ee6666abc247663d26039def3624bd30d" exitCode=0
Sep 18 20:21:43.786407 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:43.786039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h8hp2" event={"ID":"eedd0741-cae1-4c0f-8bb6-616b92088ca6","Type":"ContainerDied","Data":"e9bdf7f83a8d0e41cff9d5d6bc779e8ee6666abc247663d26039def3624bd30d"}
Sep 18 20:21:44.108304 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.108270    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76459647d6-gxxbg"]
Sep 18 20:21:44.111698 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.111682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.114663 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.114617    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b35avnu57bgkc\""
Sep 18 20:21:44.114749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.114625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 20:21:44.114953 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.114937    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cwc5j\""
Sep 18 20:21:44.116622 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.116605    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 20:21:44.116707 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.116626    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 20:21:44.116707 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.116681    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 20:21:44.116707 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.116696    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 20:21:44.131866 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.131844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76459647d6-gxxbg"]
Sep 18 20:21:44.175143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.175117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-tls\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.175243 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.175148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deb1487d-e41c-4a40-bba4-12349dcc1741-metrics-client-ca\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.175243 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.175180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmdkn\" (UniqueName: \"kubernetes.io/projected/deb1487d-e41c-4a40-bba4-12349dcc1741-kube-api-access-kmdkn\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.175328 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.175277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.175328 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.175305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.175328 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.175323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-grpc-tls\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.175498 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.175385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.175498 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.175421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.276669 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.276644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmdkn\" (UniqueName: \"kubernetes.io/projected/deb1487d-e41c-4a40-bba4-12349dcc1741-kube-api-access-kmdkn\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.276749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.276678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.276749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.276698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.276749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.276733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-grpc-tls\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.276906 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.276775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.276906 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.276795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.276906 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.276834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-tls\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.277047 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.276975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deb1487d-e41c-4a40-bba4-12349dcc1741-metrics-client-ca\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.277762 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.277729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deb1487d-e41c-4a40-bba4-12349dcc1741-metrics-client-ca\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.279365 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.279340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.279616 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.279595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.279708 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.279673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.279790 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.279769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.279837 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.279810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-grpc-tls\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.280031 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.280013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/deb1487d-e41c-4a40-bba4-12349dcc1741-secret-thanos-querier-tls\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.288883 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.288865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmdkn\" (UniqueName: \"kubernetes.io/projected/deb1487d-e41c-4a40-bba4-12349dcc1741-kube-api-access-kmdkn\") pod \"thanos-querier-76459647d6-gxxbg\" (UID: \"deb1487d-e41c-4a40-bba4-12349dcc1741\") " pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.421683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.421584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:44.664707 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.664675    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d658f7f7b-54b6n"]
Sep 18 20:21:44.668755 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.668727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.676496 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.676451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 20:21:44.686011 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.685990    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d658f7f7b-54b6n"]
Sep 18 20:21:44.737356 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.737133    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76459647d6-gxxbg"]
Sep 18 20:21:44.741692 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:44.741661    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb1487d_e41c_4a40_bba4_12349dcc1741.slice/crio-4a027bdb833a8fc98b2d9b67d27614bb17475d0f7d7da5847248686d6f291c63 WatchSource:0}: Error finding container 4a027bdb833a8fc98b2d9b67d27614bb17475d0f7d7da5847248686d6f291c63: Status 404 returned error can't find the container with id 4a027bdb833a8fc98b2d9b67d27614bb17475d0f7d7da5847248686d6f291c63
Sep 18 20:21:44.779634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.779610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-config\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.779759 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.779653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-service-ca\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.779759 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.779685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-oauth-serving-cert\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.779759 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.779741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-trusted-ca-bundle\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.779898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.779792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-serving-cert\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.779898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.779818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qrt\" (UniqueName: \"kubernetes.io/projected/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-kube-api-access-67qrt\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.779959 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.779901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-oauth-config\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.791328 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.791304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h8hp2" event={"ID":"eedd0741-cae1-4c0f-8bb6-616b92088ca6","Type":"ContainerStarted","Data":"795dda80db8b98c704f40967bd4cfadeee8f251d6741efe5b588132084fec2ec"}
Sep 18 20:21:44.791704 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.791356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h8hp2" event={"ID":"eedd0741-cae1-4c0f-8bb6-616b92088ca6","Type":"ContainerStarted","Data":"5b4dfa67ef1ce56836c98de0dea94b098d9ae572091194247a2adafab8c3e42a"}
Sep 18 20:21:44.792516 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.792496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg" event={"ID":"deb1487d-e41c-4a40-bba4-12349dcc1741","Type":"ContainerStarted","Data":"4a027bdb833a8fc98b2d9b67d27614bb17475d0f7d7da5847248686d6f291c63"}
Sep 18 20:21:44.833251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.833200    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h8hp2" podStartSLOduration=1.8882632940000001 podStartE2EDuration="2.833182014s" totalImagesPullingTime="885.149841ms" totalInitContainerRuntime="59.768879ms" isStatefulPod=true podCreationTimestamp="2026-09-18 20:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:44.832874692 +0000 UTC m=+81.642660634" watchObservedRunningTime="2026-09-18 20:21:44.833182014 +0000 UTC m=+81.642967958"
Sep 18 20:21:44.880176 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.880155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-trusted-ca-bundle\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.880274 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.880197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-serving-cert\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.880274 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.880213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qrt\" (UniqueName: \"kubernetes.io/projected/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-kube-api-access-67qrt\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.880274 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.880234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-oauth-config\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.880447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.880289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-config\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.880447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.880335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-service-ca\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.880447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.880419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-oauth-serving-cert\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.881153 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.881125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-trusted-ca-bundle\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.881255 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.881240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-config\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.881318 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.881258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-service-ca\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.881318 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.881259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-oauth-serving-cert\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.882835 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.882816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-oauth-config\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.882954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.882939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-serving-cert\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.890646 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.890619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qrt\" (UniqueName: \"kubernetes.io/projected/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-kube-api-access-67qrt\") pod \"console-6d658f7f7b-54b6n\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") " pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:44.978676 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:44.978615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:45.252322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:45.252290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d658f7f7b-54b6n"]
Sep 18 20:21:45.255111 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:45.255085    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44e28ac_33b5_48ae_9a0d_89ae49c175e2.slice/crio-bf2e0c58c14216a45e07987363aabd68fc0880a993178f85b0b81710d5fa7efb WatchSource:0}: Error finding container bf2e0c58c14216a45e07987363aabd68fc0880a993178f85b0b81710d5fa7efb: Status 404 returned error can't find the container with id bf2e0c58c14216a45e07987363aabd68fc0880a993178f85b0b81710d5fa7efb
Sep 18 20:21:45.799037 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:45.798993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d658f7f7b-54b6n" event={"ID":"b44e28ac-33b5-48ae-9a0d-89ae49c175e2","Type":"ContainerStarted","Data":"54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e"}
Sep 18 20:21:45.799506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:45.799061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d658f7f7b-54b6n" event={"ID":"b44e28ac-33b5-48ae-9a0d-89ae49c175e2","Type":"ContainerStarted","Data":"bf2e0c58c14216a45e07987363aabd68fc0880a993178f85b0b81710d5fa7efb"}
Sep 18 20:21:45.824274 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:45.824219    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d658f7f7b-54b6n" podStartSLOduration=1.8241990609999998 podStartE2EDuration="1.824199061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:45.823197835 +0000 UTC m=+82.632983776" watchObservedRunningTime="2026-09-18 20:21:45.824199061 +0000 UTC m=+82.633985001"
Sep 18 20:21:46.203834 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:46.203797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:46.204122 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:46.203881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:46.209518 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:46.209496    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:46.752924 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:46.752854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7xtmj"
Sep 18 20:21:46.806373 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:46.806345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg" event={"ID":"deb1487d-e41c-4a40-bba4-12349dcc1741","Type":"ContainerStarted","Data":"c704b8ddf8c17b53477ba9ed9c68b050c7573198f15d9fd4d0c0b4d4cb2ee22c"}
Sep 18 20:21:46.806824 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:46.806407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg" event={"ID":"deb1487d-e41c-4a40-bba4-12349dcc1741","Type":"ContainerStarted","Data":"1816624776dbc63956490c047d5d828b7900b4e93d5493fd5f25e4106d2f1901"}
Sep 18 20:21:46.806824 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:46.806435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg" event={"ID":"deb1487d-e41c-4a40-bba4-12349dcc1741","Type":"ContainerStarted","Data":"48a57376393c33a300ad6fd19cd6a190da0c5504f194f0cd19f5abcd41fe3d10"}
Sep 18 20:21:46.812061 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:46.812034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:21:47.510436 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.510396    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79959c8594-km6fw"]
Sep 18 20:21:47.513719 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.513693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.516802 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.516777    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ldkk5\""
Sep 18 20:21:47.516802 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.516790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 20:21:47.517023 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.517006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 20:21:47.517082 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.517040    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 20:21:47.517138 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.517111    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 20:21:47.517187 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.517164    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 20:21:47.522550 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.522522    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 20:21:47.533133 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.533108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79959c8594-km6fw"]
Sep 18 20:21:47.602790 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.602763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-serving-certs-ca-bundle\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.602790 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.602793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-metrics-client-ca\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.602996 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.602812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.602996 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.602950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-federate-client-tls\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.602996 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.602990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.603148 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.603029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-telemeter-client-tls\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.603148 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.603064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-secret-telemeter-client\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.603148 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.603090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvpd\" (UniqueName: \"kubernetes.io/projected/87374a97-791b-47b8-a9db-748a6504be64-kube-api-access-nnvpd\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.704369 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.704338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-serving-certs-ca-bundle\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.704369 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.704367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-metrics-client-ca\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.704640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.704388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.704640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.704427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-federate-client-tls\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.704640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.704445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.704640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.704466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-telemeter-client-tls\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.704640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.704508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-secret-telemeter-client\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.704640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.704530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvpd\" (UniqueName: \"kubernetes.io/projected/87374a97-791b-47b8-a9db-748a6504be64-kube-api-access-nnvpd\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.705212 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.705162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-metrics-client-ca\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.705386 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.705364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-serving-certs-ca-bundle\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.705424 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.705376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87374a97-791b-47b8-a9db-748a6504be64-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.707326 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.707302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-federate-client-tls\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.707438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.707422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-telemeter-client-tls\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.707517 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.707442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-secret-telemeter-client\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.707517 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.707446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87374a97-791b-47b8-a9db-748a6504be64-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.713776 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.713753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvpd\" (UniqueName: \"kubernetes.io/projected/87374a97-791b-47b8-a9db-748a6504be64-kube-api-access-nnvpd\") pod \"telemeter-client-79959c8594-km6fw\" (UID: \"87374a97-791b-47b8-a9db-748a6504be64\") " pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:47.814109 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.814083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg" event={"ID":"deb1487d-e41c-4a40-bba4-12349dcc1741","Type":"ContainerStarted","Data":"67447698913afd1b9785302444eb1c4a5460262be73b9723fb5e5cb85c864518"}
Sep 18 20:21:47.822946 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:47.822924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79959c8594-km6fw"
Sep 18 20:21:48.114495 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.114443    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79959c8594-km6fw"]
Sep 18 20:21:48.117038 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:48.117011    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87374a97_791b_47b8_a9db_748a6504be64.slice/crio-b1117c41c39c4859d3b72a76c800da2825368590447cf80bd64f8758c94f101d WatchSource:0}: Error finding container b1117c41c39c4859d3b72a76c800da2825368590447cf80bd64f8758c94f101d: Status 404 returned error can't find the container with id b1117c41c39c4859d3b72a76c800da2825368590447cf80bd64f8758c94f101d
Sep 18 20:21:48.686841 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.686809    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:21:48.690905 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.690875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.698251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.698232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 20:21:48.698448 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.698414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 20:21:48.698525 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.698488    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 20:21:48.699036 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.699017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 20:21:48.699351 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.699335    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 20:21:48.699559 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.699544    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 20:21:48.699972 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.699955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 20:21:48.700023 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.699957    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lgp99\""
Sep 18 20:21:48.700023 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.699960    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 20:21:48.700023 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.700006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 20:21:48.700247 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.700234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 20:21:48.700304 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.700292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 20:21:48.700394 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.700381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5um5a2v3pupbt\""
Sep 18 20:21:48.700794 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.700780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 20:21:48.701225 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.701210    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 20:21:48.703179 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.703159    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 20:21:48.717796 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.717769    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 20:21:48.742219 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.742181    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:21:48.813254 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813254 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gssvm\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-kube-api-access-gssvm\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813420 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813420 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813420 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813534 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813534 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813534 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813534 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813683 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813872 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813872 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813872 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.813872 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.813745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.820933 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.820900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg" event={"ID":"deb1487d-e41c-4a40-bba4-12349dcc1741","Type":"ContainerStarted","Data":"313354300eed16b2ba4edc150d967c0dc2b1eb9ef2bf8a165dc5c127fb028699"}
Sep 18 20:21:48.821324 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.820989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg" event={"ID":"deb1487d-e41c-4a40-bba4-12349dcc1741","Type":"ContainerStarted","Data":"e7c524efdf54b2f4958e473bca5c20b5f5a4050afee421202132444e806c2d7c"}
Sep 18 20:21:48.821324 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.821041    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:48.822241 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.822217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79959c8594-km6fw" event={"ID":"87374a97-791b-47b8-a9db-748a6504be64","Type":"ContainerStarted","Data":"b1117c41c39c4859d3b72a76c800da2825368590447cf80bd64f8758c94f101d"}
Sep 18 20:21:48.887704 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.887655    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg" podStartSLOduration=2.06276804 podStartE2EDuration="4.887639778s" totalImagesPullingTime="2.824871738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:48.88644919 +0000 UTC m=+85.696235159" watchObservedRunningTime="2026-09-18 20:21:48.887639778 +0000 UTC m=+85.697425720"
Sep 18 20:21:48.914763 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.914725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.914948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.914784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.914948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.914808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.914948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.914868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.914948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.914897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.914948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.914939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915194 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.914962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915194 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.914984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915194 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915194 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915194 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915194 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915194 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915194 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.915595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.915414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gssvm\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-kube-api-access-gssvm\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.916201 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.916170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.916685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.916657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.917313 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.916996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.918789 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.918445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.918887 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.918833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.919326 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.919299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.919595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.919566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.919737 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.919653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.919898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.919848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.920562 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.920227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.920562 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.920517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.920800 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.920777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.921200 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.921176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.921338 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.921238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.921571 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.921525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.921760 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.921738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.922258 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.922233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.922375 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.922358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.922570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.922553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:48.931464 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:48.931438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gssvm\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-kube-api-access-gssvm\") pod \"prometheus-k8s-0\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:49.000467 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:49.000387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:49.349289 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:49.349256    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:21:49.772361 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:21:49.772285    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c12c3b3_3c6d_416e_9e08_27c2463a190b.slice/crio-b555a3e6ed202e9f9180cd0bb63d0e89f7ec25ef6051e80c10209b3a23f6612e WatchSource:0}: Error finding container b555a3e6ed202e9f9180cd0bb63d0e89f7ec25ef6051e80c10209b3a23f6612e: Status 404 returned error can't find the container with id b555a3e6ed202e9f9180cd0bb63d0e89f7ec25ef6051e80c10209b3a23f6612e
Sep 18 20:21:49.827143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:49.827110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerStarted","Data":"b555a3e6ed202e9f9180cd0bb63d0e89f7ec25ef6051e80c10209b3a23f6612e"}
Sep 18 20:21:50.489592 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:50.489552    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d658f7f7b-54b6n"]
Sep 18 20:21:50.832492 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:50.832453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79959c8594-km6fw" event={"ID":"87374a97-791b-47b8-a9db-748a6504be64","Type":"ContainerStarted","Data":"3da718ef505cfd5fc159cd9111a0779f442fb5e1f1d8a4d2801b059de6743226"}
Sep 18 20:21:51.837617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:51.837585    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerID="3e2ddea708e9c38035f661989d926a04a5e5d76e8a7cf8a0bd2b25123ea21be8" exitCode=0
Sep 18 20:21:51.838015 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:51.837684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerDied","Data":"3e2ddea708e9c38035f661989d926a04a5e5d76e8a7cf8a0bd2b25123ea21be8"}
Sep 18 20:21:51.839997 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:51.839969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79959c8594-km6fw" event={"ID":"87374a97-791b-47b8-a9db-748a6504be64","Type":"ContainerStarted","Data":"63d773394346e14bebb2738886324e2c0eec3c224e86a55e1f8236335489df38"}
Sep 18 20:21:51.840097 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:51.840028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79959c8594-km6fw" event={"ID":"87374a97-791b-47b8-a9db-748a6504be64","Type":"ContainerStarted","Data":"3bd49cb7b78c227dff6ae5b1341ba9154de52178d93e6ea412ed86217d5e0960"}
Sep 18 20:21:51.901249 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:51.901196    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79959c8594-km6fw" podStartSLOduration=2.189643519 podStartE2EDuration="4.901177081s" totalImagesPullingTime="2.711533562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:51.89932653 +0000 UTC m=+88.709112483" watchObservedRunningTime="2026-09-18 20:21:51.901177081 +0000 UTC m=+88.710963023"
Sep 18 20:21:53.734032 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:53.734006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:54.834702 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:54.834669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76459647d6-gxxbg"
Sep 18 20:21:54.979444 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:54.979412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:21:55.864806 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:55.864771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerStarted","Data":"76198a448bf466757703720f0304ba1fac4afd4897004d6a73a6815dbbb9e84a"}
Sep 18 20:21:55.865246 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:55.864835    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerStarted","Data":"cdff77ab96fcb20995c566f040ad8458eb080eda28766634dc87aa3122698de1"}
Sep 18 20:21:55.865246 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:55.864860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerStarted","Data":"5e19159e824526ed25696e39a3b7cc74cbf3a286532a93f737f72b7d2f632145"}
Sep 18 20:21:55.865246 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:55.864883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerStarted","Data":"3ebf2973be9fa48f93ab49820feb6866cf7e399110485c32f97d9fba4ec48f7d"}
Sep 18 20:21:55.865246 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:55.864905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerStarted","Data":"31134eeeb3f58dd78a2597be500bfa7f895e700d4f718f49d17e62d568da1b57"}
Sep 18 20:21:55.865246 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:55.864926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerStarted","Data":"4719d44d7ece8636591cd31c690a6a66cc0d40424af01420c3f8993ffc6c889e"}
Sep 18 20:21:55.910052 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:55.909968    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.599516731 podStartE2EDuration="7.909950097s" totalImagesPullingTime="4.287869993s" totalInitContainerRuntime="22.563373ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:55.907163907 +0000 UTC m=+92.716949877" watchObservedRunningTime="2026-09-18 20:21:55.909950097 +0000 UTC m=+92.719736037"
Sep 18 20:21:58.746879 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:58.746821    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54747995bd-wsbr4" podUID="8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" containerName="registry" containerID="cri-o://fee39ba142083135cf7368a46fa4af62cf43b1672a221fe76974c205414ed4ea" gracePeriod=30
Sep 18 20:21:58.878297 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:58.878257    2577 generic.go:356] "Generic (PLEG): container finished" podID="8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" containerID="fee39ba142083135cf7368a46fa4af62cf43b1672a221fe76974c205414ed4ea" exitCode=0
Sep 18 20:21:58.878444 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:58.878329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54747995bd-wsbr4" event={"ID":"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6","Type":"ContainerDied","Data":"fee39ba142083135cf7368a46fa4af62cf43b1672a221fe76974c205414ed4ea"}
Sep 18 20:21:58.989966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:58.989939    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:59.001387 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.001335    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:59.099845 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.099816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-trusted-ca\") pod \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") "
Sep 18 20:21:59.100018 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.099854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-bound-sa-token\") pod \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") "
Sep 18 20:21:59.100018 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.099872    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") pod \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") "
Sep 18 20:21:59.100018 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.099905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4glm\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-kube-api-access-j4glm\") pod \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") "
Sep 18 20:21:59.100018 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.099947    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-installation-pull-secrets\") pod \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") "
Sep 18 20:21:59.100018 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.099975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-certificates\") pod \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") "
Sep 18 20:21:59.100018 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.100003    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-ca-trust-extracted\") pod \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") "
Sep 18 20:21:59.100304 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.100034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-image-registry-private-configuration\") pod \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\" (UID: \"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6\") "
Sep 18 20:21:59.100420 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.100386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:59.100776 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.100577    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-trusted-ca\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:21:59.100776 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.100584    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:59.102732 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.102708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:59.102940 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.102916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:59.103465 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.103435    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-kube-api-access-j4glm" (OuterVolumeSpecName: "kube-api-access-j4glm") pod "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6"). InnerVolumeSpecName "kube-api-access-j4glm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:59.103849 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.103828    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:59.103969 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.103945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:59.108817 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.108797    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" (UID: "8ac5fd59-a194-4364-a4f6-23da3f6a7ec6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:59.201534 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.201500    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-bound-sa-token\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:21:59.201534 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.201526    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:21:59.201534 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.201536    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4glm\" (UniqueName: \"kubernetes.io/projected/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-kube-api-access-j4glm\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:21:59.201718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.201549    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-installation-pull-secrets\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:21:59.201718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.201558    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-registry-certificates\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:21:59.201718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.201566    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-ca-trust-extracted\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:21:59.201718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.201575    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6-image-registry-private-configuration\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:21:59.883575 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.883543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54747995bd-wsbr4" event={"ID":"8ac5fd59-a194-4364-a4f6-23da3f6a7ec6","Type":"ContainerDied","Data":"1c39e4055429318f41bca3ab2b2f15797e63270eee7f79d946bd2f6a1c573853"}
Sep 18 20:21:59.883979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.883604    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54747995bd-wsbr4"
Sep 18 20:21:59.883979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.883614    2577 scope.go:117] "RemoveContainer" containerID="fee39ba142083135cf7368a46fa4af62cf43b1672a221fe76974c205414ed4ea"
Sep 18 20:21:59.901011 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.900989    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54747995bd-wsbr4"]
Sep 18 20:21:59.907077 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:21:59.907052    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54747995bd-wsbr4"]
Sep 18 20:22:01.242195 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:01.242164    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" path="/var/lib/kubelet/pods/8ac5fd59-a194-4364-a4f6-23da3f6a7ec6/volumes"
Sep 18 20:22:06.748688 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:06.748658    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hptqd"
Sep 18 20:22:13.293785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:13.293752    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559cdf5c69-75fdj"]
Sep 18 20:22:15.514731 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.514695    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d658f7f7b-54b6n" podUID="b44e28ac-33b5-48ae-9a0d-89ae49c175e2" containerName="console" containerID="cri-o://54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e" gracePeriod=15
Sep 18 20:22:15.756117 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.756095    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:22:15.836727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.836702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-oauth-serving-cert\") pod \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") "
Sep 18 20:22:15.836858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.836736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-trusted-ca-bundle\") pod \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") "
Sep 18 20:22:15.836858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.836756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-oauth-config\") pod \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") "
Sep 18 20:22:15.836858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.836790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-config\") pod \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") "
Sep 18 20:22:15.836858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.836821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-service-ca\") pod \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") "
Sep 18 20:22:15.836858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.836844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67qrt\" (UniqueName: \"kubernetes.io/projected/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-kube-api-access-67qrt\") pod \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") "
Sep 18 20:22:15.837101 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.836916    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-serving-cert\") pod \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\" (UID: \"b44e28ac-33b5-48ae-9a0d-89ae49c175e2\") "
Sep 18 20:22:15.837216 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.837138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b44e28ac-33b5-48ae-9a0d-89ae49c175e2" (UID: "b44e28ac-33b5-48ae-9a0d-89ae49c175e2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.837216 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.837146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b44e28ac-33b5-48ae-9a0d-89ae49c175e2" (UID: "b44e28ac-33b5-48ae-9a0d-89ae49c175e2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.837499 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.837439    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-service-ca" (OuterVolumeSpecName: "service-ca") pod "b44e28ac-33b5-48ae-9a0d-89ae49c175e2" (UID: "b44e28ac-33b5-48ae-9a0d-89ae49c175e2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.837499 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.837494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-config" (OuterVolumeSpecName: "console-config") pod "b44e28ac-33b5-48ae-9a0d-89ae49c175e2" (UID: "b44e28ac-33b5-48ae-9a0d-89ae49c175e2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.839024 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.839002    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-kube-api-access-67qrt" (OuterVolumeSpecName: "kube-api-access-67qrt") pod "b44e28ac-33b5-48ae-9a0d-89ae49c175e2" (UID: "b44e28ac-33b5-48ae-9a0d-89ae49c175e2"). InnerVolumeSpecName "kube-api-access-67qrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:15.839114 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.839040    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b44e28ac-33b5-48ae-9a0d-89ae49c175e2" (UID: "b44e28ac-33b5-48ae-9a0d-89ae49c175e2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:15.839114 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.839057    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b44e28ac-33b5-48ae-9a0d-89ae49c175e2" (UID: "b44e28ac-33b5-48ae-9a0d-89ae49c175e2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:15.937449 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.937411    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-oauth-serving-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.937449 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.937446    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-trusted-ca-bundle\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.937449 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.937455    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-oauth-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.937634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.937464    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.937634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.937492    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-service-ca\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.937634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.937500    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qrt\" (UniqueName: \"kubernetes.io/projected/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-kube-api-access-67qrt\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.937634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.937508    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b44e28ac-33b5-48ae-9a0d-89ae49c175e2-console-serving-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.948150 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.948123    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44e28ac-33b5-48ae-9a0d-89ae49c175e2" containerID="54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e" exitCode=0
Sep 18 20:22:15.948256 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.948154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d658f7f7b-54b6n" event={"ID":"b44e28ac-33b5-48ae-9a0d-89ae49c175e2","Type":"ContainerDied","Data":"54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e"}
Sep 18 20:22:15.948256 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.948174    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d658f7f7b-54b6n"
Sep 18 20:22:15.948256 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.948213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d658f7f7b-54b6n" event={"ID":"b44e28ac-33b5-48ae-9a0d-89ae49c175e2","Type":"ContainerDied","Data":"bf2e0c58c14216a45e07987363aabd68fc0880a993178f85b0b81710d5fa7efb"}
Sep 18 20:22:15.948256 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.948232    2577 scope.go:117] "RemoveContainer" containerID="54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e"
Sep 18 20:22:15.956179 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.956069    2577 scope.go:117] "RemoveContainer" containerID="54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e"
Sep 18 20:22:15.956410 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:22:15.956374    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e\": container with ID starting with 54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e not found: ID does not exist" containerID="54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e"
Sep 18 20:22:15.956496 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.956412    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e"} err="failed to get container status \"54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e\": rpc error: code = NotFound desc = could not find container \"54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e\": container with ID starting with 54cc44f681e3ca8d970ba975c27262aa4e970ceda6fe2f79bf98bde14762f71e not found: ID does not exist"
Sep 18 20:22:15.971678 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.971653    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d658f7f7b-54b6n"]
Sep 18 20:22:15.977766 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:15.977745    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d658f7f7b-54b6n"]
Sep 18 20:22:17.242409 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:17.242378    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44e28ac-33b5-48ae-9a0d-89ae49c175e2" path="/var/lib/kubelet/pods/b44e28ac-33b5-48ae-9a0d-89ae49c175e2/volumes"
Sep 18 20:22:38.312360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.312314    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-559cdf5c69-75fdj" podUID="b07762b6-9c63-4cb4-a3c3-38d94dc004c6" containerName="console" containerID="cri-o://8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7" gracePeriod=15
Sep 18 20:22:38.550434 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.550412    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:22:38.610747 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.610717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-service-ca\") pod \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") "
Sep 18 20:22:38.610747 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.610749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-oauth-config\") pod \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") "
Sep 18 20:22:38.610984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.610780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-serving-cert\") pod \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") "
Sep 18 20:22:38.610984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.610800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-oauth-serving-cert\") pod \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") "
Sep 18 20:22:38.610984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.610840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsh6t\" (UniqueName: \"kubernetes.io/projected/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-kube-api-access-hsh6t\") pod \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") "
Sep 18 20:22:38.610984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.610889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-config\") pod \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\" (UID: \"b07762b6-9c63-4cb4-a3c3-38d94dc004c6\") "
Sep 18 20:22:38.611243 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.611138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-service-ca" (OuterVolumeSpecName: "service-ca") pod "b07762b6-9c63-4cb4-a3c3-38d94dc004c6" (UID: "b07762b6-9c63-4cb4-a3c3-38d94dc004c6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:38.611304 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.611259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b07762b6-9c63-4cb4-a3c3-38d94dc004c6" (UID: "b07762b6-9c63-4cb4-a3c3-38d94dc004c6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:38.611304 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.611287    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-config" (OuterVolumeSpecName: "console-config") pod "b07762b6-9c63-4cb4-a3c3-38d94dc004c6" (UID: "b07762b6-9c63-4cb4-a3c3-38d94dc004c6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:38.612941 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.612916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b07762b6-9c63-4cb4-a3c3-38d94dc004c6" (UID: "b07762b6-9c63-4cb4-a3c3-38d94dc004c6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:38.613032 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.613008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b07762b6-9c63-4cb4-a3c3-38d94dc004c6" (UID: "b07762b6-9c63-4cb4-a3c3-38d94dc004c6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:38.613137 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.613113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-kube-api-access-hsh6t" (OuterVolumeSpecName: "kube-api-access-hsh6t") pod "b07762b6-9c63-4cb4-a3c3-38d94dc004c6" (UID: "b07762b6-9c63-4cb4-a3c3-38d94dc004c6"). InnerVolumeSpecName "kube-api-access-hsh6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:38.712175 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.712134    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-serving-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:38.712175 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.712170    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-oauth-serving-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:38.712175 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.712178    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsh6t\" (UniqueName: \"kubernetes.io/projected/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-kube-api-access-hsh6t\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:38.712175 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.712187    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:38.712175 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.712195    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-service-ca\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:38.712446 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:38.712203    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07762b6-9c63-4cb4-a3c3-38d94dc004c6-console-oauth-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:22:39.037812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.037724    2577 generic.go:356] "Generic (PLEG): container finished" podID="b07762b6-9c63-4cb4-a3c3-38d94dc004c6" containerID="8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7" exitCode=0
Sep 18 20:22:39.037812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.037767    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559cdf5c69-75fdj"
Sep 18 20:22:39.037812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.037772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559cdf5c69-75fdj" event={"ID":"b07762b6-9c63-4cb4-a3c3-38d94dc004c6","Type":"ContainerDied","Data":"8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7"}
Sep 18 20:22:39.038068 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.037856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559cdf5c69-75fdj" event={"ID":"b07762b6-9c63-4cb4-a3c3-38d94dc004c6","Type":"ContainerDied","Data":"0589702caf9a9f8d3e7971c6eb880bfd0898ca1fb46923edd8345bf1d25d4a09"}
Sep 18 20:22:39.038068 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.037884    2577 scope.go:117] "RemoveContainer" containerID="8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7"
Sep 18 20:22:39.046171 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.046155    2577 scope.go:117] "RemoveContainer" containerID="8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7"
Sep 18 20:22:39.046435 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:22:39.046416    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7\": container with ID starting with 8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7 not found: ID does not exist" containerID="8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7"
Sep 18 20:22:39.046539 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.046447    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7"} err="failed to get container status \"8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7\": rpc error: code = NotFound desc = could not find container \"8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7\": container with ID starting with 8987ce805365fbfa1f88a00c724d036a082daaa6f07c522e4750e146e7aa03b7 not found: ID does not exist"
Sep 18 20:22:39.057666 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.057644    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559cdf5c69-75fdj"]
Sep 18 20:22:39.061836 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.061813    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-559cdf5c69-75fdj"]
Sep 18 20:22:39.243309 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:39.243277    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b07762b6-9c63-4cb4-a3c3-38d94dc004c6" path="/var/lib/kubelet/pods/b07762b6-9c63-4cb4-a3c3-38d94dc004c6/volumes"
Sep 18 20:22:49.001542 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:49.001515    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:22:49.019839 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:49.019811    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:22:49.087030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:22:49.087005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:06.925287 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:06.925254    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:06.925713 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:06.925532    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="prometheus" containerID="cri-o://4719d44d7ece8636591cd31c690a6a66cc0d40424af01420c3f8993ffc6c889e" gracePeriod=600
Sep 18 20:23:06.925713 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:06.925566    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="thanos-sidecar" containerID="cri-o://3ebf2973be9fa48f93ab49820feb6866cf7e399110485c32f97d9fba4ec48f7d" gracePeriod=600
Sep 18 20:23:06.925713 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:06.925594    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://76198a448bf466757703720f0304ba1fac4afd4897004d6a73a6815dbbb9e84a" gracePeriod=600
Sep 18 20:23:06.925713 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:06.925620    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="config-reloader" containerID="cri-o://31134eeeb3f58dd78a2597be500bfa7f895e700d4f718f49d17e62d568da1b57" gracePeriod=600
Sep 18 20:23:06.925713 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:06.925566    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy" containerID="cri-o://cdff77ab96fcb20995c566f040ad8458eb080eda28766634dc87aa3122698de1" gracePeriod=600
Sep 18 20:23:06.925713 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:06.925625    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy-web" containerID="cri-o://5e19159e824526ed25696e39a3b7cc74cbf3a286532a93f737f72b7d2f632145" gracePeriod=600
Sep 18 20:23:07.136961 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.136931    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerID="76198a448bf466757703720f0304ba1fac4afd4897004d6a73a6815dbbb9e84a" exitCode=0
Sep 18 20:23:07.136961 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.136954    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerID="cdff77ab96fcb20995c566f040ad8458eb080eda28766634dc87aa3122698de1" exitCode=0
Sep 18 20:23:07.136961 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.136960    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerID="5e19159e824526ed25696e39a3b7cc74cbf3a286532a93f737f72b7d2f632145" exitCode=0
Sep 18 20:23:07.136961 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.136965    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerID="3ebf2973be9fa48f93ab49820feb6866cf7e399110485c32f97d9fba4ec48f7d" exitCode=0
Sep 18 20:23:07.136961 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.136971    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerID="31134eeeb3f58dd78a2597be500bfa7f895e700d4f718f49d17e62d568da1b57" exitCode=0
Sep 18 20:23:07.137321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.136976    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerID="4719d44d7ece8636591cd31c690a6a66cc0d40424af01420c3f8993ffc6c889e" exitCode=0
Sep 18 20:23:07.137321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.137000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerDied","Data":"76198a448bf466757703720f0304ba1fac4afd4897004d6a73a6815dbbb9e84a"}
Sep 18 20:23:07.137321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.137066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerDied","Data":"cdff77ab96fcb20995c566f040ad8458eb080eda28766634dc87aa3122698de1"}
Sep 18 20:23:07.137321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.137093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerDied","Data":"5e19159e824526ed25696e39a3b7cc74cbf3a286532a93f737f72b7d2f632145"}
Sep 18 20:23:07.137321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.137116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerDied","Data":"3ebf2973be9fa48f93ab49820feb6866cf7e399110485c32f97d9fba4ec48f7d"}
Sep 18 20:23:07.137321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.137139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerDied","Data":"31134eeeb3f58dd78a2597be500bfa7f895e700d4f718f49d17e62d568da1b57"}
Sep 18 20:23:07.137321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.137161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerDied","Data":"4719d44d7ece8636591cd31c690a6a66cc0d40424af01420c3f8993ffc6c889e"}
Sep 18 20:23:07.164224 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.164206    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:07.324529 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gssvm\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-kube-api-access-gssvm\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324664 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324539    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-tls-assets\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324664 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324664 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324576    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-web-config\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-1\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-serving-certs-ca-bundle\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-2\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324788    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-trusted-ca-bundle\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.324916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324874    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-metrics-client-certs\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.325875 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.324906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-tls\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.325971 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.325902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-grpc-tls\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.325971 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.325940    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-0\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.326068 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.325990    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-db\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.326068 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326021    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config-out\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.326068 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326056    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-kube-rbac-proxy\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.326200 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-kubelet-serving-ca-bundle\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.326200 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-thanos-prometheus-http-client-file\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.326200 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-metrics-client-ca\") pod \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\" (UID: \"8c12c3b3-3c6d-416e-9e08-27c2463a190b\") "
Sep 18 20:23:07.326357 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:07.327549 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:07.327549 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326675    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:07.327549 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.326901    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:07.328127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.328092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:07.329505 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.328555    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:07.329505 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.329067    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:07.329930 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.329902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:07.330022 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.329966    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.330081 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330023    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-kube-api-access-gssvm" (OuterVolumeSpecName: "kube-api-access-gssvm") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "kube-api-access-gssvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:07.330081 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config" (OuterVolumeSpecName: "config") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.330543 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330494    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-metrics-client-certs\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330543 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330521    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330543 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330539    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330733 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330564    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-metrics-client-ca\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330733 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330580    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gssvm\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-kube-api-access-gssvm\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330733 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330595    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c12c3b3-3c6d-416e-9e08-27c2463a190b-tls-assets\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330733 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330608    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330733 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330625    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330733 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330668    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330733 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330685    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.330733 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330700    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.331129 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.330751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:07.333124 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.332931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.333124 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.333005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.333254 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.333221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.333312 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.333273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.333370 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.333307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config-out" (OuterVolumeSpecName: "config-out") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:07.333564 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.333537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.336311 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.336281    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.340851 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.340833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-web-config" (OuterVolumeSpecName: "web-config") pod "8c12c3b3-3c6d-416e-9e08-27c2463a190b" (UID: "8c12c3b3-3c6d-416e-9e08-27c2463a190b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:07.431300 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431267    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-prometheus-k8s-db\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.431300 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431299    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c12c3b3-3c6d-416e-9e08-27c2463a190b-config-out\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.431300 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431308    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-kube-rbac-proxy\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.431454 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431317    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.431454 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431327    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-web-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.431454 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431335    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.431454 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431344    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.431454 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431352    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:07.431454 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:07.431360    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c12c3b3-3c6d-416e-9e08-27c2463a190b-secret-grpc-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:23:08.143418 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.143386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c12c3b3-3c6d-416e-9e08-27c2463a190b","Type":"ContainerDied","Data":"b555a3e6ed202e9f9180cd0bb63d0e89f7ec25ef6051e80c10209b3a23f6612e"}
Sep 18 20:23:08.143884 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.143437    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.143884 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.143500    2577 scope.go:117] "RemoveContainer" containerID="76198a448bf466757703720f0304ba1fac4afd4897004d6a73a6815dbbb9e84a"
Sep 18 20:23:08.151441 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.151426    2577 scope.go:117] "RemoveContainer" containerID="cdff77ab96fcb20995c566f040ad8458eb080eda28766634dc87aa3122698de1"
Sep 18 20:23:08.158299 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.158278    2577 scope.go:117] "RemoveContainer" containerID="5e19159e824526ed25696e39a3b7cc74cbf3a286532a93f737f72b7d2f632145"
Sep 18 20:23:08.164427 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.164409    2577 scope.go:117] "RemoveContainer" containerID="3ebf2973be9fa48f93ab49820feb6866cf7e399110485c32f97d9fba4ec48f7d"
Sep 18 20:23:08.168133 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.168109    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:08.171406 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.171388    2577 scope.go:117] "RemoveContainer" containerID="31134eeeb3f58dd78a2597be500bfa7f895e700d4f718f49d17e62d568da1b57"
Sep 18 20:23:08.172752 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.172733    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:08.177576 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.177557    2577 scope.go:117] "RemoveContainer" containerID="4719d44d7ece8636591cd31c690a6a66cc0d40424af01420c3f8993ffc6c889e"
Sep 18 20:23:08.184379 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.184361    2577 scope.go:117] "RemoveContainer" containerID="3e2ddea708e9c38035f661989d926a04a5e5d76e8a7cf8a0bd2b25123ea21be8"
Sep 18 20:23:08.200677 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200656    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:08.200900 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200888    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy-thanos"
Sep 18 20:23:08.200951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200902    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy-thanos"
Sep 18 20:23:08.200951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200915    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy"
Sep 18 20:23:08.200951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200921    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy"
Sep 18 20:23:08.200951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200930    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="thanos-sidecar"
Sep 18 20:23:08.200951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200935    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="thanos-sidecar"
Sep 18 20:23:08.200951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200941    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="prometheus"
Sep 18 20:23:08.200951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200946    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="prometheus"
Sep 18 20:23:08.200951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200952    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy-web"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200958    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy-web"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200965    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="init-config-reloader"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200970    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="init-config-reloader"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200977    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b07762b6-9c63-4cb4-a3c3-38d94dc004c6" containerName="console"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200982    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b07762b6-9c63-4cb4-a3c3-38d94dc004c6" containerName="console"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200988    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44e28ac-33b5-48ae-9a0d-89ae49c175e2" containerName="console"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.200993    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44e28ac-33b5-48ae-9a0d-89ae49c175e2" containerName="console"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201001    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="config-reloader"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201006    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="config-reloader"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201012    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" containerName="registry"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201017    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" containerName="registry"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201077    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="prometheus"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201084    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="config-reloader"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201091    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac5fd59-a194-4364-a4f6-23da3f6a7ec6" containerName="registry"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201098    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy-web"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201103    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b07762b6-9c63-4cb4-a3c3-38d94dc004c6" containerName="console"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201108    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="thanos-sidecar"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201113    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201117    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" containerName="kube-rbac-proxy-thanos"
Sep 18 20:23:08.201162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.201122    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44e28ac-33b5-48ae-9a0d-89ae49c175e2" containerName="console"
Sep 18 20:23:08.205969 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.205953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.208639 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.208615    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 20:23:08.209045 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 20:23:08.209133 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 20:23:08.209133 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209025    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 20:23:08.209243 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209116    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 20:23:08.209243 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209207    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 20:23:08.209417 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209402    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5um5a2v3pupbt\""
Sep 18 20:23:08.209507 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209492    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 20:23:08.209563 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 20:23:08.209797 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lgp99\""
Sep 18 20:23:08.209864 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209810    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 20:23:08.209864 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209842    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 20:23:08.209948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209924    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 20:23:08.209948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209935    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 20:23:08.210029 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.209957    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 20:23:08.212640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.212619    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 20:23:08.214676 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.214658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 20:23:08.223884 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.223865    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:08.338528 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0786959-e2d1-4d36-8517-dc2ccb452333-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwwm\" (UniqueName: \"kubernetes.io/projected/c0786959-e2d1-4d36-8517-dc2ccb452333-kube-api-access-vhwwm\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338835 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338835 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338835 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338835 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.338835 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339014 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339014 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339014 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339014 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-config\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339014 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339014 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.338986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339014 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.339014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339208 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.339038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0786959-e2d1-4d36-8517-dc2ccb452333-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.339208 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.339055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439745 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439745 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439944 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439944 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-config\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439944 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439944 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439944 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439944 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0786959-e2d1-4d36-8517-dc2ccb452333-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.439944 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.439993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0786959-e2d1-4d36-8517-dc2ccb452333-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwwm\" (UniqueName: \"kubernetes.io/projected/c0786959-e2d1-4d36-8517-dc2ccb452333-kube-api-access-vhwwm\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440283 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440876 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440876 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.440876 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.440853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.441565 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.441538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.442769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.442050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.443110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.443001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.443282 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.443257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-config\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.443399 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.443313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.443651 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.443629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0786959-e2d1-4d36-8517-dc2ccb452333-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.443764 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.443742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.443990 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.443955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.444112 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.444089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0786959-e2d1-4d36-8517-dc2ccb452333-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.444827 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.444799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.445579 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.445550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.445652 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.445557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.445689 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.445662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.445754 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.445733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0786959-e2d1-4d36-8517-dc2ccb452333-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.445915 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.445900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.446947 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.446932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0786959-e2d1-4d36-8517-dc2ccb452333-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.450264 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.450248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwwm\" (UniqueName: \"kubernetes.io/projected/c0786959-e2d1-4d36-8517-dc2ccb452333-kube-api-access-vhwwm\") pod \"prometheus-k8s-0\" (UID: \"c0786959-e2d1-4d36-8517-dc2ccb452333\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.515339 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.515319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:08.795747 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:08.795722    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:08.797789 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:23:08.797761    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0786959_e2d1_4d36_8517_dc2ccb452333.slice/crio-b3e4c5d321569d001f95145e6a8454d20f450db1e6606882cb7e45b6c4fed104 WatchSource:0}: Error finding container b3e4c5d321569d001f95145e6a8454d20f450db1e6606882cb7e45b6c4fed104: Status 404 returned error can't find the container with id b3e4c5d321569d001f95145e6a8454d20f450db1e6606882cb7e45b6c4fed104
Sep 18 20:23:09.148818 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:09.148785    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0786959-e2d1-4d36-8517-dc2ccb452333" containerID="8311c2ee4dec5fc0cf92018d17786a756debdadc47aec62069b45d61df1d1ca2" exitCode=0
Sep 18 20:23:09.149243 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:09.148852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0786959-e2d1-4d36-8517-dc2ccb452333","Type":"ContainerDied","Data":"8311c2ee4dec5fc0cf92018d17786a756debdadc47aec62069b45d61df1d1ca2"}
Sep 18 20:23:09.149243 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:09.148896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0786959-e2d1-4d36-8517-dc2ccb452333","Type":"ContainerStarted","Data":"b3e4c5d321569d001f95145e6a8454d20f450db1e6606882cb7e45b6c4fed104"}
Sep 18 20:23:09.249266 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:09.248991    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c12c3b3-3c6d-416e-9e08-27c2463a190b" path="/var/lib/kubelet/pods/8c12c3b3-3c6d-416e-9e08-27c2463a190b/volumes"
Sep 18 20:23:10.155376 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:10.155340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0786959-e2d1-4d36-8517-dc2ccb452333","Type":"ContainerStarted","Data":"e99cf00d21430e7abf1950fbe9051175ecdd6fe910acf167d6cbdd894ff8bb0c"}
Sep 18 20:23:10.155770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:10.155410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0786959-e2d1-4d36-8517-dc2ccb452333","Type":"ContainerStarted","Data":"a466bb7378a73d5cbc36b3a5005f049514750f320b66385d4b9ddecafed37cfa"}
Sep 18 20:23:10.155770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:10.155437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0786959-e2d1-4d36-8517-dc2ccb452333","Type":"ContainerStarted","Data":"1dfe758dd04a951926a5ced4fb7eee5c57e14269ea28827042697e331399e5c0"}
Sep 18 20:23:10.155770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:10.155461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0786959-e2d1-4d36-8517-dc2ccb452333","Type":"ContainerStarted","Data":"1cf06719ad0a46426937164da3f0c1313e750f566858445f16e95339dbd93de5"}
Sep 18 20:23:10.155770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:10.155506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0786959-e2d1-4d36-8517-dc2ccb452333","Type":"ContainerStarted","Data":"cf69f088719564ae01bd9e9c6bde7e66448494d15edb0c0a1b79e63670149268"}
Sep 18 20:23:10.155770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:10.155530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0786959-e2d1-4d36-8517-dc2ccb452333","Type":"ContainerStarted","Data":"ed7467943c30caad599c9fce7db0b564c5eec18945e2333e0e790f469fa988dc"}
Sep 18 20:23:10.188178 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:10.188099    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.170185511 podStartE2EDuration="2.188079725s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.894214ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:10.183632432 +0000 UTC m=+166.993418374" watchObservedRunningTime="2026-09-18 20:23:10.188079725 +0000 UTC m=+166.997865668"
Sep 18 20:23:13.515857 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:23:13.515826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:24:08.516984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:24:08.515720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:24:08.540972 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:24:08.540946    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:24:09.371995 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:24:09.371970    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:25:25.172133 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:25:25.172102    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 20:26:09.634646 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.634555    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"]
Sep 18 20:26:09.637938 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.637917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:09.641878 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.641853    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 20:26:09.642011 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.641970    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 20:26:09.642078 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.642050    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 20:26:09.642879 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.642856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-cwz6l\""
Sep 18 20:26:09.649013 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.648953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"]
Sep 18 20:26:09.780166 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.780135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edc1147e-f3cf-4387-a4ec-3ac1e529f050-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-77rbj\" (UID: \"edc1147e-f3cf-4387-a4ec-3ac1e529f050\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:09.780356 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.780177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgbw\" (UniqueName: \"kubernetes.io/projected/edc1147e-f3cf-4387-a4ec-3ac1e529f050-kube-api-access-2tgbw\") pod \"llmisvc-controller-manager-645f9c8fd6-77rbj\" (UID: \"edc1147e-f3cf-4387-a4ec-3ac1e529f050\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:09.880678 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.880633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgbw\" (UniqueName: \"kubernetes.io/projected/edc1147e-f3cf-4387-a4ec-3ac1e529f050-kube-api-access-2tgbw\") pod \"llmisvc-controller-manager-645f9c8fd6-77rbj\" (UID: \"edc1147e-f3cf-4387-a4ec-3ac1e529f050\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:09.880856 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.880737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edc1147e-f3cf-4387-a4ec-3ac1e529f050-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-77rbj\" (UID: \"edc1147e-f3cf-4387-a4ec-3ac1e529f050\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:09.880919 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:26:09.880856    2577 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 20:26:09.880966 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:26:09.880924    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc1147e-f3cf-4387-a4ec-3ac1e529f050-cert podName:edc1147e-f3cf-4387-a4ec-3ac1e529f050 nodeName:}" failed. No retries permitted until 2026-09-18 20:26:10.38090283 +0000 UTC m=+347.190688755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/edc1147e-f3cf-4387-a4ec-3ac1e529f050-cert") pod "llmisvc-controller-manager-645f9c8fd6-77rbj" (UID: "edc1147e-f3cf-4387-a4ec-3ac1e529f050") : secret "llmisvc-webhook-server-cert" not found
Sep 18 20:26:09.889141 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:09.889078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgbw\" (UniqueName: \"kubernetes.io/projected/edc1147e-f3cf-4387-a4ec-3ac1e529f050-kube-api-access-2tgbw\") pod \"llmisvc-controller-manager-645f9c8fd6-77rbj\" (UID: \"edc1147e-f3cf-4387-a4ec-3ac1e529f050\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:10.385425 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:10.385391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edc1147e-f3cf-4387-a4ec-3ac1e529f050-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-77rbj\" (UID: \"edc1147e-f3cf-4387-a4ec-3ac1e529f050\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:10.387827 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:10.387803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edc1147e-f3cf-4387-a4ec-3ac1e529f050-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-77rbj\" (UID: \"edc1147e-f3cf-4387-a4ec-3ac1e529f050\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:10.549435 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:10.549386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:10.816240 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:10.816166    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"]
Sep 18 20:26:10.820564 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:26:10.820540    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podedc1147e_f3cf_4387_a4ec_3ac1e529f050.slice/crio-3194080b94474d8c050bc524a191295180684c542188d2c2a918ca02500313b6 WatchSource:0}: Error finding container 3194080b94474d8c050bc524a191295180684c542188d2c2a918ca02500313b6: Status 404 returned error can't find the container with id 3194080b94474d8c050bc524a191295180684c542188d2c2a918ca02500313b6
Sep 18 20:26:10.822168 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:10.822151    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:26:11.783652 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:11.783616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj" event={"ID":"edc1147e-f3cf-4387-a4ec-3ac1e529f050","Type":"ContainerStarted","Data":"3194080b94474d8c050bc524a191295180684c542188d2c2a918ca02500313b6"}
Sep 18 20:26:13.792352 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:13.792317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj" event={"ID":"edc1147e-f3cf-4387-a4ec-3ac1e529f050","Type":"ContainerStarted","Data":"9c9da3eb8e0848ccc6dc6472c4bf19cffa95ae5eb60dbc92b6d1d85a3a4289dc"}
Sep 18 20:26:13.792736 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:13.792444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:26:13.809088 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:13.809039    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj" podStartSLOduration=2.503177414 podStartE2EDuration="4.809024333s" totalImagesPullingTime="2.305846919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:13.808044008 +0000 UTC m=+350.617829974" watchObservedRunningTime="2026-09-18 20:26:13.809024333 +0000 UTC m=+350.618810274"
Sep 18 20:26:44.800973 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:26:44.800938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-77rbj"
Sep 18 20:27:44.514940 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.514863    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bp7r8"]
Sep 18 20:27:44.518068 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.518052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bp7r8"
Sep 18 20:27:44.521564 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.521544    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 20:27:44.521751 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.521738    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wgp8j\""
Sep 18 20:27:44.522247 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.522229    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 20:27:44.533594 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.533568    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bp7r8"]
Sep 18 20:27:44.579955 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.579929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9rr\" (UniqueName: \"kubernetes.io/projected/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-kube-api-access-gr9rr\") pod \"s3-init-bp7r8\" (UID: \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\") " pod="kserve/s3-init-bp7r8"
Sep 18 20:27:44.580043 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.579984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-model-cache\") pod \"s3-init-bp7r8\" (UID: \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\") " pod="kserve/s3-init-bp7r8"
Sep 18 20:27:44.681169 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.681146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-model-cache\") pod \"s3-init-bp7r8\" (UID: \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\") " pod="kserve/s3-init-bp7r8"
Sep 18 20:27:44.681243 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.681181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9rr\" (UniqueName: \"kubernetes.io/projected/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-kube-api-access-gr9rr\") pod \"s3-init-bp7r8\" (UID: \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\") " pod="kserve/s3-init-bp7r8"
Sep 18 20:27:44.681502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.681465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-model-cache\") pod \"s3-init-bp7r8\" (UID: \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\") " pod="kserve/s3-init-bp7r8"
Sep 18 20:27:44.690270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.690243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9rr\" (UniqueName: \"kubernetes.io/projected/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-kube-api-access-gr9rr\") pod \"s3-init-bp7r8\" (UID: \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\") " pod="kserve/s3-init-bp7r8"
Sep 18 20:27:44.843130 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:44.843105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bp7r8"
Sep 18 20:27:45.122413 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:45.122385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bp7r8"]
Sep 18 20:27:45.124231 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:27:45.124211    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7ff47d2_eeec_4b7a_a6f4_06693d2c58c8.slice/crio-20577bbe14069406aa26e747a1775824af3ea5906804b40c74b840025c4c8836 WatchSource:0}: Error finding container 20577bbe14069406aa26e747a1775824af3ea5906804b40c74b840025c4c8836: Status 404 returned error can't find the container with id 20577bbe14069406aa26e747a1775824af3ea5906804b40c74b840025c4c8836
Sep 18 20:27:46.122443 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:46.122411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7r8" event={"ID":"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8","Type":"ContainerStarted","Data":"20577bbe14069406aa26e747a1775824af3ea5906804b40c74b840025c4c8836"}
Sep 18 20:27:49.135701 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:49.135667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7r8" event={"ID":"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8","Type":"ContainerStarted","Data":"21f6357da49d17416de44b26ecfbf95b06ff234e1fde0caef0e2b16e90ad8b73"}
Sep 18 20:27:51.144322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:51.144242    2577 generic.go:356] "Generic (PLEG): container finished" podID="d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" containerID="21f6357da49d17416de44b26ecfbf95b06ff234e1fde0caef0e2b16e90ad8b73" exitCode=0
Sep 18 20:27:51.144322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:51.144292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7r8" event={"ID":"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8","Type":"ContainerDied","Data":"21f6357da49d17416de44b26ecfbf95b06ff234e1fde0caef0e2b16e90ad8b73"}
Sep 18 20:27:56.165864 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:56.165830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7r8" event={"ID":"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8","Type":"ContainerStarted","Data":"3f1e3be09d793135219e28e446a793712ea5c1182bd8af01f7031dc9b540422d"}
Sep 18 20:27:56.186538 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:27:56.186462    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-bp7r8" podStartSLOduration=2.116372487 podStartE2EDuration="12.18644455s" totalImagesPullingTime="8.27381845s" totalInitContainerRuntime="1.796253613s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:56.185316617 +0000 UTC m=+452.995102578" watchObservedRunningTime="2026-09-18 20:27:56.18644455 +0000 UTC m=+452.996230493"
Sep 18 20:29:01.402502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:01.402447    2577 generic.go:356] "Generic (PLEG): container finished" podID="d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" containerID="3f1e3be09d793135219e28e446a793712ea5c1182bd8af01f7031dc9b540422d" exitCode=0
Sep 18 20:29:01.402991 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:01.402526    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7r8" event={"ID":"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8","Type":"ContainerDied","Data":"3f1e3be09d793135219e28e446a793712ea5c1182bd8af01f7031dc9b540422d"}
Sep 18 20:29:02.527586 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:02.527564    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bp7r8"
Sep 18 20:29:02.579591 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:02.579567    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-model-cache\") pod \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\" (UID: \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\") "
Sep 18 20:29:02.579734 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:02.579606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr9rr\" (UniqueName: \"kubernetes.io/projected/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-kube-api-access-gr9rr\") pod \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\" (UID: \"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8\") "
Sep 18 20:29:02.581688 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:02.581666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-kube-api-access-gr9rr" (OuterVolumeSpecName: "kube-api-access-gr9rr") pod "d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" (UID: "d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8"). InnerVolumeSpecName "kube-api-access-gr9rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:29:02.601698 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:02.601671    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" (UID: "d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:02.680292 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:02.680226    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-model-cache\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:29:02.680292 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:02.680250    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9rr\" (UniqueName: \"kubernetes.io/projected/d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8-kube-api-access-gr9rr\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:29:03.411216 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:03.411183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7r8" event={"ID":"d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8","Type":"ContainerDied","Data":"20577bbe14069406aa26e747a1775824af3ea5906804b40c74b840025c4c8836"}
Sep 18 20:29:03.411370 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:03.411224    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20577bbe14069406aa26e747a1775824af3ea5906804b40c74b840025c4c8836"
Sep 18 20:29:03.411370 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:03.411198    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bp7r8"
Sep 18 20:29:15.789027 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.788946    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"]
Sep 18 20:29:15.789391 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.789354    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" containerName="s3-init"
Sep 18 20:29:15.789391 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.789368    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" containerName="s3-init"
Sep 18 20:29:15.789521 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.789404    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" containerName="download-hf-model"
Sep 18 20:29:15.789521 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.789412    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" containerName="download-hf-model"
Sep 18 20:29:15.789521 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.789486    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8" containerName="s3-init"
Sep 18 20:29:15.793089 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.793073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.795875 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.795849    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ttgfx\""
Sep 18 20:29:15.796025 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.795849    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:29:15.796025 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.795905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-d48f1-predictor-serving-cert\""
Sep 18 20:29:15.796025 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.795930    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:29:15.796025 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.795940    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:29:15.796271 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.796253    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-d48f1-kube-rbac-proxy-sar-config\""
Sep 18 20:29:15.796271 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.796266    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:29:15.807443 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.807422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"]
Sep 18 20:29:15.871687 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.871655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6km9k\" (UniqueName: \"kubernetes.io/projected/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-api-access-6km9k\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.871855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.871747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0825e48c-e485-4b4b-9f7f-47b195eb5138-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.871855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.871816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.871966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.871868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.871966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.871908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0825e48c-e485-4b4b-9f7f-47b195eb5138-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.887575 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.887546    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"]
Sep 18 20:29:15.890963 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.890947    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:15.893438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.893413    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 18 20:29:15.893674 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.893655    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 20:29:15.903900 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.903876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"]
Sep 18 20:29:15.972406 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6km9k\" (UniqueName: \"kubernetes.io/projected/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-api-access-6km9k\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.972406 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc0552a-dffe-4d30-9a85-a65368822525-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:15.972680 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzj4j\" (UniqueName: \"kubernetes.io/projected/8bc0552a-dffe-4d30-9a85-a65368822525-kube-api-access-mzj4j\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:15.972680 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0825e48c-e485-4b4b-9f7f-47b195eb5138-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.972680 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:15.972680 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.972680 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.972915 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0825e48c-e485-4b4b-9f7f-47b195eb5138-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.972915 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:15.972915 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:15.973087 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.972895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0825e48c-e485-4b4b-9f7f-47b195eb5138-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.973267 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.973243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.973383 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.973303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.975167 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.975147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0825e48c-e485-4b4b-9f7f-47b195eb5138-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.985221 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.985193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6km9k\" (UniqueName: \"kubernetes.io/projected/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-api-access-6km9k\") pod \"isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:15.989894 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.989872    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"]
Sep 18 20:29:15.994150 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.994133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:15.996920 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:15.996904    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:29:16.007535 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.007515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"]
Sep 18 20:29:16.073545 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzj4j\" (UniqueName: \"kubernetes.io/projected/8bc0552a-dffe-4d30-9a85-a65368822525-kube-api-access-mzj4j\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.073722 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.073722 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.073722 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.073722 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c17bff-6a09-4b74-af8f-3ba6150cd567-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.073930 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.073930 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.073930 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.074081 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc0552a-dffe-4d30-9a85-a65368822525-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.074081 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.073975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnfsm\" (UniqueName: \"kubernetes.io/projected/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-api-access-jnfsm\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.074081 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:29:16.074028    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 18 20:29:16.074228 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:29:16.074093    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls podName:8bc0552a-dffe-4d30-9a85-a65368822525 nodeName:}" failed. No retries permitted until 2026-09-18 20:29:16.574076124 +0000 UTC m=+533.383862047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls") pod "isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" (UID: "8bc0552a-dffe-4d30-9a85-a65368822525") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 18 20:29:16.074295 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.074264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc0552a-dffe-4d30-9a85-a65368822525-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.074571 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.074548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.074670 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.074604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.084589 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.084565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzj4j\" (UniqueName: \"kubernetes.io/projected/8bc0552a-dffe-4d30-9a85-a65368822525-kube-api-access-mzj4j\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.105669 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.105646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:16.175522 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.175442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnfsm\" (UniqueName: \"kubernetes.io/projected/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-api-access-jnfsm\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.175522 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.175507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.175729 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.175538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.175729 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.175577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.175729 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.175611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c17bff-6a09-4b74-af8f-3ba6150cd567-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.176202 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.176128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.176497 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.176455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.176611 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.176545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.178011 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.177992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c17bff-6a09-4b74-af8f-3ba6150cd567-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.183001 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.182977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnfsm\" (UniqueName: \"kubernetes.io/projected/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-api-access-jnfsm\") pod \"isvc-canary-rollback-predictor-7f7b644c9-45drl\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.303990 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.303948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:16.382588 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.382563    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"]
Sep 18 20:29:16.384795 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:29:16.384768    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0825e48c_e485_4b4b_9f7f_47b195eb5138.slice/crio-ea1c4e86045559bfff3f5f90a67eb84de37df3b09833c3e8876668ba393e8e9f WatchSource:0}: Error finding container ea1c4e86045559bfff3f5f90a67eb84de37df3b09833c3e8876668ba393e8e9f: Status 404 returned error can't find the container with id ea1c4e86045559bfff3f5f90a67eb84de37df3b09833c3e8876668ba393e8e9f
Sep 18 20:29:16.464925 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.464890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerStarted","Data":"6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502"}
Sep 18 20:29:16.465045 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.464981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerStarted","Data":"ea1c4e86045559bfff3f5f90a67eb84de37df3b09833c3e8876668ba393e8e9f"}
Sep 18 20:29:16.575506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.575461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"]
Sep 18 20:29:16.577674 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:29:16.577650    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c17bff_6a09_4b74_af8f_3ba6150cd567.slice/crio-3375a708424a11f29672abd367a9e9784880c7d12cf3e54e1d3bb1aa64cb70a9 WatchSource:0}: Error finding container 3375a708424a11f29672abd367a9e9784880c7d12cf3e54e1d3bb1aa64cb70a9: Status 404 returned error can't find the container with id 3375a708424a11f29672abd367a9e9784880c7d12cf3e54e1d3bb1aa64cb70a9
Sep 18 20:29:16.578785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.578761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.581298 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.581279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:16.802609 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:16.802563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:17.086440 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:17.086405    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"]
Sep 18 20:29:17.090030 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:29:17.090004    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc0552a_dffe_4d30_9a85_a65368822525.slice/crio-e4b4149127351283decf4615705f56cd9fa176c53a89e300f19ec2e054f8efbf WatchSource:0}: Error finding container e4b4149127351283decf4615705f56cd9fa176c53a89e300f19ec2e054f8efbf: Status 404 returned error can't find the container with id e4b4149127351283decf4615705f56cd9fa176c53a89e300f19ec2e054f8efbf
Sep 18 20:29:17.471144 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:17.471062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" event={"ID":"8bc0552a-dffe-4d30-9a85-a65368822525","Type":"ContainerStarted","Data":"c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf"}
Sep 18 20:29:17.471144 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:17.471128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" event={"ID":"8bc0552a-dffe-4d30-9a85-a65368822525","Type":"ContainerStarted","Data":"e4b4149127351283decf4615705f56cd9fa176c53a89e300f19ec2e054f8efbf"}
Sep 18 20:29:17.472812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:17.472785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" event={"ID":"a4c17bff-6a09-4b74-af8f-3ba6150cd567","Type":"ContainerStarted","Data":"e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9"}
Sep 18 20:29:17.472940 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:17.472838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" event={"ID":"a4c17bff-6a09-4b74-af8f-3ba6150cd567","Type":"ContainerStarted","Data":"3375a708424a11f29672abd367a9e9784880c7d12cf3e54e1d3bb1aa64cb70a9"}
Sep 18 20:29:20.491342 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:20.491312    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerID="e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9" exitCode=0
Sep 18 20:29:20.491764 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:20.491397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" event={"ID":"a4c17bff-6a09-4b74-af8f-3ba6150cd567","Type":"ContainerDied","Data":"e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9"}
Sep 18 20:29:20.493053 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:20.492983    2577 generic.go:356] "Generic (PLEG): container finished" podID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerID="6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502" exitCode=0
Sep 18 20:29:20.493053 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:20.493013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerDied","Data":"6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502"}
Sep 18 20:29:21.509504 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:21.509002    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bc0552a-dffe-4d30-9a85-a65368822525" containerID="c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf" exitCode=0
Sep 18 20:29:21.509504 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:21.509093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" event={"ID":"8bc0552a-dffe-4d30-9a85-a65368822525","Type":"ContainerDied","Data":"c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf"}
Sep 18 20:29:40.674596 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:40.674557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" event={"ID":"8bc0552a-dffe-4d30-9a85-a65368822525","Type":"ContainerStarted","Data":"ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f"}
Sep 18 20:29:40.678106 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:40.678080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" event={"ID":"a4c17bff-6a09-4b74-af8f-3ba6150cd567","Type":"ContainerStarted","Data":"06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b"}
Sep 18 20:29:40.680966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:40.680890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerStarted","Data":"346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901"}
Sep 18 20:29:43.697407 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.697366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" event={"ID":"8bc0552a-dffe-4d30-9a85-a65368822525","Type":"ContainerStarted","Data":"3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f"}
Sep 18 20:29:43.697887 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.697593    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:43.697887 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.697771    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:43.699368 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.699327    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:29:43.700198 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.700011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" event={"ID":"a4c17bff-6a09-4b74-af8f-3ba6150cd567","Type":"ContainerStarted","Data":"fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396"}
Sep 18 20:29:43.700198 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.700066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:43.700198 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.700115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:43.701394 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.701369    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:29:43.702643 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.702621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerStarted","Data":"c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e"}
Sep 18 20:29:43.719470 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.719282    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podStartSLOduration=3.497385273 podStartE2EDuration="28.71926549s" totalImagesPullingTime="21.37724353s" totalInitContainerRuntime="3.844636687s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:43.718556919 +0000 UTC m=+560.528342861" watchObservedRunningTime="2026-09-18 20:29:43.71926549 +0000 UTC m=+560.529051433"
Sep 18 20:29:43.739811 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:43.739757    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podStartSLOduration=2.674981496 podStartE2EDuration="28.739739433s" totalImagesPullingTime="22.372433022s" totalInitContainerRuntime="3.692324915s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:43.739613116 +0000 UTC m=+560.549399059" watchObservedRunningTime="2026-09-18 20:29:43.739739433 +0000 UTC m=+560.549525379"
Sep 18 20:29:44.707657 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:44.707607    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:29:44.708094 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:44.707801    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:29:45.714325 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:45.714283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerStarted","Data":"44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d"}
Sep 18 20:29:45.714830 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:45.714577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:45.714830 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:45.714734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:45.715805 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:45.715780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:29:45.738400 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:45.738348    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podStartSLOduration=2.24199535 podStartE2EDuration="30.738334568s" totalImagesPullingTime="24.655279362s" totalInitContainerRuntime="3.841059856s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:45.736529811 +0000 UTC m=+562.546315753" watchObservedRunningTime="2026-09-18 20:29:45.738334568 +0000 UTC m=+562.548120510"
Sep 18 20:29:46.720151 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:46.720117    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:29:46.720623 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:46.720288    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:46.721515 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:46.721467    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:29:47.724814 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:47.724772    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:29:47.725219 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:47.725199    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:29:49.714708 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:49.714677    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:29:49.715179 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:49.714775    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:29:49.715248 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:49.715192    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:29:49.715559 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:49.715528    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:29:52.728974 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:52.728946    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:29:52.729518 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:52.729492    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:29:52.729727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:52.729706    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:29:59.716302 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:59.716260    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:29:59.716302 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:29:59.716271    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:30:02.730258 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:02.730163    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:30:02.730701 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:02.730601    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:09.715760 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:09.715713    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:30:09.716237 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:09.715767    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:30:12.729515 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:12.729466    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:30:12.729923 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:12.729733    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:19.715542 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:19.715493    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:30:19.715956 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:19.715494    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:30:22.730013 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:22.729972    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:30:22.730583 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:22.730555    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:29.715300 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:29.715252    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:30:29.715910 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:29.715428    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:30:32.730341 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:32.730304    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:30:32.730825 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:32.730799    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:39.715392 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:39.715294    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:30:39.715994 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:39.715495    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:30:42.729571 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:42.729527    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:30:42.730050 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:42.730026    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:49.715388 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:49.715348    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:30:49.716000 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:49.715571    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:30:52.730113 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:52.730065    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:30:52.730581 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:52.730516    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:59.715522 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:59.715454    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 20:30:59.716073 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:59.716051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:30:59.842290 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:59.842257    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"]
Sep 18 20:30:59.842535 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:59.842502    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container" containerID="cri-o://06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b" gracePeriod=30
Sep 18 20:30:59.842676 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:59.842561    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kube-rbac-proxy" containerID="cri-o://fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396" gracePeriod=30
Sep 18 20:30:59.843358 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:30:59.843291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:31:00.023490 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:00.023419    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerID="fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396" exitCode=2
Sep 18 20:31:00.023595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:00.023511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" event={"ID":"a4c17bff-6a09-4b74-af8f-3ba6150cd567","Type":"ContainerDied","Data":"fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396"}
Sep 18 20:31:02.730669 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:02.730632    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:31:02.732966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:02.730727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:31:04.380133 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.380109    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:31:04.406764 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.406736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-cabundle-cert\") pod \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") "
Sep 18 20:31:04.406877 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.406782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-rbac-proxy-sar-config\") pod \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") "
Sep 18 20:31:04.406954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.406934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kserve-provision-location\") pod \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") "
Sep 18 20:31:04.406999 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.406971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnfsm\" (UniqueName: \"kubernetes.io/projected/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-api-access-jnfsm\") pod \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") "
Sep 18 20:31:04.407095 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.407080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c17bff-6a09-4b74-af8f-3ba6150cd567-proxy-tls\") pod \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\" (UID: \"a4c17bff-6a09-4b74-af8f-3ba6150cd567\") "
Sep 18 20:31:04.407349 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.407085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4c17bff-6a09-4b74-af8f-3ba6150cd567" (UID: "a4c17bff-6a09-4b74-af8f-3ba6150cd567"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:04.407349 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.407202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4c17bff-6a09-4b74-af8f-3ba6150cd567" (UID: "a4c17bff-6a09-4b74-af8f-3ba6150cd567"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:04.407349 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.407204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4c17bff-6a09-4b74-af8f-3ba6150cd567" (UID: "a4c17bff-6a09-4b74-af8f-3ba6150cd567"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:04.407349 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.407325    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:04.407349 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.407338    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:04.407349 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.407347    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:04.409119 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.409093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-api-access-jnfsm" (OuterVolumeSpecName: "kube-api-access-jnfsm") pod "a4c17bff-6a09-4b74-af8f-3ba6150cd567" (UID: "a4c17bff-6a09-4b74-af8f-3ba6150cd567"). InnerVolumeSpecName "kube-api-access-jnfsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:04.409198 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.409163    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c17bff-6a09-4b74-af8f-3ba6150cd567-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4c17bff-6a09-4b74-af8f-3ba6150cd567" (UID: "a4c17bff-6a09-4b74-af8f-3ba6150cd567"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:31:04.508434 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.508383    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c17bff-6a09-4b74-af8f-3ba6150cd567-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:04.508434 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:04.508404    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnfsm\" (UniqueName: \"kubernetes.io/projected/a4c17bff-6a09-4b74-af8f-3ba6150cd567-kube-api-access-jnfsm\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:05.043386 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.043352    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerID="06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b" exitCode=0
Sep 18 20:31:05.043574 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.043405    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"
Sep 18 20:31:05.043574 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.043437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" event={"ID":"a4c17bff-6a09-4b74-af8f-3ba6150cd567","Type":"ContainerDied","Data":"06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b"}
Sep 18 20:31:05.043574 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.043539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl" event={"ID":"a4c17bff-6a09-4b74-af8f-3ba6150cd567","Type":"ContainerDied","Data":"3375a708424a11f29672abd367a9e9784880c7d12cf3e54e1d3bb1aa64cb70a9"}
Sep 18 20:31:05.043574 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.043568    2577 scope.go:117] "RemoveContainer" containerID="fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396"
Sep 18 20:31:05.052004 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.051990    2577 scope.go:117] "RemoveContainer" containerID="06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b"
Sep 18 20:31:05.059417 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.059399    2577 scope.go:117] "RemoveContainer" containerID="e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9"
Sep 18 20:31:05.065214 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.065193    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"]
Sep 18 20:31:05.066393 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.066379    2577 scope.go:117] "RemoveContainer" containerID="fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396"
Sep 18 20:31:05.066658 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:05.066634    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396\": container with ID starting with fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396 not found: ID does not exist" containerID="fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396"
Sep 18 20:31:05.066709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.066662    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396"} err="failed to get container status \"fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396\": rpc error: code = NotFound desc = could not find container \"fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396\": container with ID starting with fc715acb7b2d7ae5a43ad55e874caa8f1eb3b7bdfd657b40cbc2a8a05b13d396 not found: ID does not exist"
Sep 18 20:31:05.066709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.066678    2577 scope.go:117] "RemoveContainer" containerID="06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b"
Sep 18 20:31:05.066932 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:05.066908    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b\": container with ID starting with 06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b not found: ID does not exist" containerID="06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b"
Sep 18 20:31:05.067007 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.066940    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b"} err="failed to get container status \"06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b\": rpc error: code = NotFound desc = could not find container \"06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b\": container with ID starting with 06bbe764f6cb64793d53e13b4b06fcf85dcf6d1eaf8b12b26c0a920bf35f019b not found: ID does not exist"
Sep 18 20:31:05.067007 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.066962    2577 scope.go:117] "RemoveContainer" containerID="e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9"
Sep 18 20:31:05.067215 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:05.067200    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9\": container with ID starting with e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9 not found: ID does not exist" containerID="e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9"
Sep 18 20:31:05.067255 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.067218    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9"} err="failed to get container status \"e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9\": rpc error: code = NotFound desc = could not find container \"e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9\": container with ID starting with e687cb37a7724e5b22d7799c6ecf8dbf8cd84ff299a254b1db3be25d406680c9 not found: ID does not exist"
Sep 18 20:31:05.069266 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.069248    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-45drl"]
Sep 18 20:31:05.242447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.242414    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" path="/var/lib/kubelet/pods/a4c17bff-6a09-4b74-af8f-3ba6150cd567/volumes"
Sep 18 20:31:05.639092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.639064    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"]
Sep 18 20:31:05.639528 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.639284    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" containerID="cri-o://ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f" gracePeriod=30
Sep 18 20:31:05.639528 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.639343    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kube-rbac-proxy" containerID="cri-o://3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f" gracePeriod=30
Sep 18 20:31:05.720816 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.720783    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"]
Sep 18 20:31:05.721116 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.721100    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kube-rbac-proxy"
Sep 18 20:31:05.721116 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.721117    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kube-rbac-proxy"
Sep 18 20:31:05.721261 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.721130    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="storage-initializer"
Sep 18 20:31:05.721261 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.721137    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="storage-initializer"
Sep 18 20:31:05.721261 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.721159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container"
Sep 18 20:31:05.721261 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.721164    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container"
Sep 18 20:31:05.721261 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.721206    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kube-rbac-proxy"
Sep 18 20:31:05.721261 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.721214    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c17bff-6a09-4b74-af8f-3ba6150cd567" containerName="kserve-container"
Sep 18 20:31:05.725686 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.725669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.730289 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.730269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:31:05.737966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.737946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"]
Sep 18 20:31:05.818305 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.818276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.818447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.818318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.818447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.818392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.818447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.818415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.818447 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.818433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rql9g\" (UniqueName: \"kubernetes.io/projected/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-api-access-rql9g\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.919383 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.919299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.919383 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.919362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.919610 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.919385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rql9g\" (UniqueName: \"kubernetes.io/projected/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-api-access-rql9g\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.919610 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.919441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.919610 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.919511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.919797 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.919775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.920090 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.920061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.920090 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.920084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.921886 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.921868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:05.929063 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:05.929044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rql9g\" (UniqueName: \"kubernetes.io/projected/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-api-access-rql9g\") pod \"isvc-canary-rollback-predictor-7f7b644c9-89s7r\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:06.035219 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:06.035165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:06.050164 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:06.050140    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bc0552a-dffe-4d30-9a85-a65368822525" containerID="3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f" exitCode=2
Sep 18 20:31:06.050267 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:06.050203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" event={"ID":"8bc0552a-dffe-4d30-9a85-a65368822525","Type":"ContainerDied","Data":"3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f"}
Sep 18 20:31:06.308206 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:06.308178    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"]
Sep 18 20:31:06.309876 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:31:06.309844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf6d415_5bf2_4d9b_9a94_0f36235d3140.slice/crio-242b79e04c41b710f261c3811aa9d401fa6e514a689cf44db7f5b601b16e26bd WatchSource:0}: Error finding container 242b79e04c41b710f261c3811aa9d401fa6e514a689cf44db7f5b601b16e26bd: Status 404 returned error can't find the container with id 242b79e04c41b710f261c3811aa9d401fa6e514a689cf44db7f5b601b16e26bd
Sep 18 20:31:07.056878 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:07.056844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" event={"ID":"5bf6d415-5bf2-4d9b-9a94-0f36235d3140","Type":"ContainerStarted","Data":"0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe"}
Sep 18 20:31:07.057211 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:07.056931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" event={"ID":"5bf6d415-5bf2-4d9b-9a94-0f36235d3140","Type":"ContainerStarted","Data":"242b79e04c41b710f261c3811aa9d401fa6e514a689cf44db7f5b601b16e26bd"}
Sep 18 20:31:09.708359 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:09.708319    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 18 20:31:09.715843 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:09.715805    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 18 20:31:10.271842 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.271816    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:31:10.372691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.372614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-kube-rbac-proxy-sar-config\") pod \"8bc0552a-dffe-4d30-9a85-a65368822525\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") "
Sep 18 20:31:10.372691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.372674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls\") pod \"8bc0552a-dffe-4d30-9a85-a65368822525\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") "
Sep 18 20:31:10.372900 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.372703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzj4j\" (UniqueName: \"kubernetes.io/projected/8bc0552a-dffe-4d30-9a85-a65368822525-kube-api-access-mzj4j\") pod \"8bc0552a-dffe-4d30-9a85-a65368822525\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") "
Sep 18 20:31:10.372900 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.372730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-cabundle-cert\") pod \"8bc0552a-dffe-4d30-9a85-a65368822525\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") "
Sep 18 20:31:10.372900 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.372748    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc0552a-dffe-4d30-9a85-a65368822525-kserve-provision-location\") pod \"8bc0552a-dffe-4d30-9a85-a65368822525\" (UID: \"8bc0552a-dffe-4d30-9a85-a65368822525\") "
Sep 18 20:31:10.373110 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.372996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bc0552a-dffe-4d30-9a85-a65368822525" (UID: "8bc0552a-dffe-4d30-9a85-a65368822525"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:10.373217 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.373074    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bc0552a-dffe-4d30-9a85-a65368822525" (UID: "8bc0552a-dffe-4d30-9a85-a65368822525"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:10.373217 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.373151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc0552a-dffe-4d30-9a85-a65368822525-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bc0552a-dffe-4d30-9a85-a65368822525" (UID: "8bc0552a-dffe-4d30-9a85-a65368822525"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:10.374852 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.374829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bc0552a-dffe-4d30-9a85-a65368822525" (UID: "8bc0552a-dffe-4d30-9a85-a65368822525"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:31:10.374852 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.374840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc0552a-dffe-4d30-9a85-a65368822525-kube-api-access-mzj4j" (OuterVolumeSpecName: "kube-api-access-mzj4j") pod "8bc0552a-dffe-4d30-9a85-a65368822525" (UID: "8bc0552a-dffe-4d30-9a85-a65368822525"). InnerVolumeSpecName "kube-api-access-mzj4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:10.473996 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.473959    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:10.473996 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.473991    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bc0552a-dffe-4d30-9a85-a65368822525-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:10.473996 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.474001    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bc0552a-dffe-4d30-9a85-a65368822525-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:10.474228 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.474011    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bc0552a-dffe-4d30-9a85-a65368822525-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:10.474228 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:10.474020    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzj4j\" (UniqueName: \"kubernetes.io/projected/8bc0552a-dffe-4d30-9a85-a65368822525-kube-api-access-mzj4j\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:11.051189 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.051152    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"]
Sep 18 20:31:11.051610 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.051501    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" containerID="cri-o://346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901" gracePeriod=30
Sep 18 20:31:11.051610 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.051522    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" containerID="cri-o://44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d" gracePeriod=30
Sep 18 20:31:11.051781 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.051532    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy" containerID="cri-o://c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e" gracePeriod=30
Sep 18 20:31:11.073613 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.073588    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bc0552a-dffe-4d30-9a85-a65368822525" containerID="ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f" exitCode=0
Sep 18 20:31:11.073727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.073639    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"
Sep 18 20:31:11.073727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.073669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" event={"ID":"8bc0552a-dffe-4d30-9a85-a65368822525","Type":"ContainerDied","Data":"ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f"}
Sep 18 20:31:11.073824 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.073737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r" event={"ID":"8bc0552a-dffe-4d30-9a85-a65368822525","Type":"ContainerDied","Data":"e4b4149127351283decf4615705f56cd9fa176c53a89e300f19ec2e054f8efbf"}
Sep 18 20:31:11.073824 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.073759    2577 scope.go:117] "RemoveContainer" containerID="3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f"
Sep 18 20:31:11.075296 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.075272    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerID="0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe" exitCode=0
Sep 18 20:31:11.075356 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.075315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" event={"ID":"5bf6d415-5bf2-4d9b-9a94-0f36235d3140","Type":"ContainerDied","Data":"0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe"}
Sep 18 20:31:11.114036 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.114014    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"]
Sep 18 20:31:11.119726 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.119703    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-qp48r"]
Sep 18 20:31:11.164728 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.164556    2577 scope.go:117] "RemoveContainer" containerID="ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f"
Sep 18 20:31:11.172513 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.172489    2577 scope.go:117] "RemoveContainer" containerID="c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf"
Sep 18 20:31:11.188107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.188083    2577 scope.go:117] "RemoveContainer" containerID="3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f"
Sep 18 20:31:11.188421 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:11.188397    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f\": container with ID starting with 3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f not found: ID does not exist" containerID="3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f"
Sep 18 20:31:11.188513 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.188431    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f"} err="failed to get container status \"3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f\": rpc error: code = NotFound desc = could not find container \"3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f\": container with ID starting with 3a134f3048bff1d80b95e8da9768c992b941623928cad6def75211ebb3bab11f not found: ID does not exist"
Sep 18 20:31:11.188513 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.188458    2577 scope.go:117] "RemoveContainer" containerID="ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f"
Sep 18 20:31:11.188794 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:11.188771    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f\": container with ID starting with ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f not found: ID does not exist" containerID="ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f"
Sep 18 20:31:11.188858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.188799    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f"} err="failed to get container status \"ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f\": rpc error: code = NotFound desc = could not find container \"ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f\": container with ID starting with ad2863fb10d5837a3965da8758c553a1405abeb1913d33250c8a3a3603d42c8f not found: ID does not exist"
Sep 18 20:31:11.188858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.188815    2577 scope.go:117] "RemoveContainer" containerID="c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf"
Sep 18 20:31:11.189086 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:11.189066    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf\": container with ID starting with c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf not found: ID does not exist" containerID="c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf"
Sep 18 20:31:11.189151 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.189089    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf"} err="failed to get container status \"c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf\": rpc error: code = NotFound desc = could not find container \"c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf\": container with ID starting with c45297d8d4b562e1be521148df27c4b54a971200c7d33e47a04261671131febf not found: ID does not exist"
Sep 18 20:31:11.244210 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.244182    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" path="/var/lib/kubelet/pods/8bc0552a-dffe-4d30-9a85-a65368822525/volumes"
Sep 18 20:31:11.351539 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351510    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"]
Sep 18 20:31:11.351833 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351820    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="storage-initializer"
Sep 18 20:31:11.351898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351835    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="storage-initializer"
Sep 18 20:31:11.351898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351850    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container"
Sep 18 20:31:11.351898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351859    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container"
Sep 18 20:31:11.351898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351872    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kube-rbac-proxy"
Sep 18 20:31:11.351898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351878    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kube-rbac-proxy"
Sep 18 20:31:11.352132 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351928    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kube-rbac-proxy"
Sep 18 20:31:11.352132 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.351938    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc0552a-dffe-4d30-9a85-a65368822525" containerName="kserve-container"
Sep 18 20:31:11.354785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.354758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.358269 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.358249    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-12fdc-kube-rbac-proxy-sar-config\""
Sep 18 20:31:11.358629 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.358613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-12fdc-predictor-serving-cert\""
Sep 18 20:31:11.369002 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.368974    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"]
Sep 18 20:31:11.484495 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.484443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f17e442f-ac76-493b-a93f-18334ff6354a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.484685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.484533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.484685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.484563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh6qq\" (UniqueName: \"kubernetes.io/projected/f17e442f-ac76-493b-a93f-18334ff6354a-kube-api-access-vh6qq\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.484685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.484621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.484685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.484644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f17e442f-ac76-493b-a93f-18334ff6354a-proxy-tls\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.585525 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.585495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.585525 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.585527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh6qq\" (UniqueName: \"kubernetes.io/projected/f17e442f-ac76-493b-a93f-18334ff6354a-kube-api-access-vh6qq\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.585739 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.585658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.585739 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.585707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f17e442f-ac76-493b-a93f-18334ff6354a-proxy-tls\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.585837 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.585747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f17e442f-ac76-493b-a93f-18334ff6354a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.586145 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.586119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f17e442f-ac76-493b-a93f-18334ff6354a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.586233 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.586214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.586345 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.586325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.588167 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.588136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f17e442f-ac76-493b-a93f-18334ff6354a-proxy-tls\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.593966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.593945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh6qq\" (UniqueName: \"kubernetes.io/projected/f17e442f-ac76-493b-a93f-18334ff6354a-kube-api-access-vh6qq\") pod \"isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.665078 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.665022    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:11.936568 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:11.936544    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"]
Sep 18 20:31:11.938544 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:31:11.938516    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17e442f_ac76_493b_a93f_18334ff6354a.slice/crio-0dd8196b174d1366a83b860f4572a5637e14cd9aece1626473bae1db438ac5e7 WatchSource:0}: Error finding container 0dd8196b174d1366a83b860f4572a5637e14cd9aece1626473bae1db438ac5e7: Status 404 returned error can't find the container with id 0dd8196b174d1366a83b860f4572a5637e14cd9aece1626473bae1db438ac5e7
Sep 18 20:31:12.083101 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.083067    2577 generic.go:356] "Generic (PLEG): container finished" podID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerID="c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e" exitCode=2
Sep 18 20:31:12.083666 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.083137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerDied","Data":"c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e"}
Sep 18 20:31:12.085381 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.085362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" event={"ID":"5bf6d415-5bf2-4d9b-9a94-0f36235d3140","Type":"ContainerStarted","Data":"9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749"}
Sep 18 20:31:12.085485 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.085419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" event={"ID":"5bf6d415-5bf2-4d9b-9a94-0f36235d3140","Type":"ContainerStarted","Data":"8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266"}
Sep 18 20:31:12.085654 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.085638    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:12.086970 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.086953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" event={"ID":"f17e442f-ac76-493b-a93f-18334ff6354a","Type":"ContainerStarted","Data":"e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff"}
Sep 18 20:31:12.087064 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.086991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" event={"ID":"f17e442f-ac76-493b-a93f-18334ff6354a","Type":"ContainerStarted","Data":"0dd8196b174d1366a83b860f4572a5637e14cd9aece1626473bae1db438ac5e7"}
Sep 18 20:31:12.105840 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.105674    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podStartSLOduration=3.285243197 podStartE2EDuration="7.105659519s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820416322s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:12.104358461 +0000 UTC m=+648.914144404" watchObservedRunningTime="2026-09-18 20:31:12.105659519 +0000 UTC m=+648.915445462"
Sep 18 20:31:12.725426 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.725388    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 18 20:31:12.729823 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.729780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:31:12.730079 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:12.730061    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:13.091704 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:13.091672    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:13.092916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:13.092887    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:31:14.094840 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:14.094806    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:31:16.104626 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:16.104588    2577 generic.go:356] "Generic (PLEG): container finished" podID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerID="346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901" exitCode=0
Sep 18 20:31:16.105013 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:16.104657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerDied","Data":"346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901"}
Sep 18 20:31:16.106050 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:16.106028    2577 generic.go:356] "Generic (PLEG): container finished" podID="f17e442f-ac76-493b-a93f-18334ff6354a" containerID="e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff" exitCode=0
Sep 18 20:31:16.106182 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:16.106090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" event={"ID":"f17e442f-ac76-493b-a93f-18334ff6354a","Type":"ContainerDied","Data":"e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff"}
Sep 18 20:31:16.107375 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:16.107360    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:31:17.725133 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:17.725084    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 18 20:31:19.100295 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:19.100259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:31:19.100869 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:19.100834    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:31:22.725396 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:22.725351    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 18 20:31:22.725922 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:22.725523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:31:22.729586 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:22.729548    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:31:22.729754 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:22.729726    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:27.725641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:27.725596    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 18 20:31:29.101323 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:29.101286    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:31:32.725233 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:32.725192    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 18 20:31:32.729536 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:32.729503    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 18 20:31:32.729681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:32.729639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:31:32.729888 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:32.729861    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:32.730014 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:32.729999    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:31:36.193270 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:36.193234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" event={"ID":"f17e442f-ac76-493b-a93f-18334ff6354a","Type":"ContainerStarted","Data":"6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109"}
Sep 18 20:31:36.193643 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:36.193301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" event={"ID":"f17e442f-ac76-493b-a93f-18334ff6354a","Type":"ContainerStarted","Data":"7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919"}
Sep 18 20:31:36.193643 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:36.193534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:36.213493 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:36.213437    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podStartSLOduration=1.9639693249999999 podStartE2EDuration="25.213426582s" totalImagesPullingTime="19.47504023s" totalInitContainerRuntime="3.774417027s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:36.213271987 +0000 UTC m=+673.023057929" watchObservedRunningTime="2026-09-18 20:31:36.213426582 +0000 UTC m=+673.023212524"
Sep 18 20:31:37.197594 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:37.197565    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:37.198951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:37.198920    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:31:37.725456 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:37.725417    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 18 20:31:38.201745 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:38.201708    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:31:39.101105 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:39.101070    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:31:41.189311 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.189287    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:31:41.217428 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.217399    2577 generic.go:356] "Generic (PLEG): container finished" podID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerID="44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d" exitCode=0
Sep 18 20:31:41.217567 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.217463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerDied","Data":"44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d"}
Sep 18 20:31:41.217567 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.217495    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"
Sep 18 20:31:41.217641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.217590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws" event={"ID":"0825e48c-e485-4b4b-9f7f-47b195eb5138","Type":"ContainerDied","Data":"ea1c4e86045559bfff3f5f90a67eb84de37df3b09833c3e8876668ba393e8e9f"}
Sep 18 20:31:41.217682 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.217641    2577 scope.go:117] "RemoveContainer" containerID="44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d"
Sep 18 20:31:41.227020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.227005    2577 scope.go:117] "RemoveContainer" containerID="c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e"
Sep 18 20:31:41.231218 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.231193    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0825e48c-e485-4b4b-9f7f-47b195eb5138-proxy-tls\") pod \"0825e48c-e485-4b4b-9f7f-47b195eb5138\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") "
Sep 18 20:31:41.231311 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.231238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6km9k\" (UniqueName: \"kubernetes.io/projected/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-api-access-6km9k\") pod \"0825e48c-e485-4b4b-9f7f-47b195eb5138\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") "
Sep 18 20:31:41.231311 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.231276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0825e48c-e485-4b4b-9f7f-47b195eb5138-kserve-provision-location\") pod \"0825e48c-e485-4b4b-9f7f-47b195eb5138\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") "
Sep 18 20:31:41.231424 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.231312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-cabundle-cert\") pod \"0825e48c-e485-4b4b-9f7f-47b195eb5138\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") "
Sep 18 20:31:41.231424 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.231342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-rbac-proxy-sar-config\") pod \"0825e48c-e485-4b4b-9f7f-47b195eb5138\" (UID: \"0825e48c-e485-4b4b-9f7f-47b195eb5138\") "
Sep 18 20:31:41.231660 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.231629    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0825e48c-e485-4b4b-9f7f-47b195eb5138-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0825e48c-e485-4b4b-9f7f-47b195eb5138" (UID: "0825e48c-e485-4b4b-9f7f-47b195eb5138"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:41.231774 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.231716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0825e48c-e485-4b4b-9f7f-47b195eb5138" (UID: "0825e48c-e485-4b4b-9f7f-47b195eb5138"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:41.231846 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.231794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0825e48c-e485-4b4b-9f7f-47b195eb5138" (UID: "0825e48c-e485-4b4b-9f7f-47b195eb5138"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:41.233219 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.233197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0825e48c-e485-4b4b-9f7f-47b195eb5138-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0825e48c-e485-4b4b-9f7f-47b195eb5138" (UID: "0825e48c-e485-4b4b-9f7f-47b195eb5138"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:31:41.233763 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.233731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-api-access-6km9k" (OuterVolumeSpecName: "kube-api-access-6km9k") pod "0825e48c-e485-4b4b-9f7f-47b195eb5138" (UID: "0825e48c-e485-4b4b-9f7f-47b195eb5138"). InnerVolumeSpecName "kube-api-access-6km9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:41.235394 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.235381    2577 scope.go:117] "RemoveContainer" containerID="346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901"
Sep 18 20:31:41.247745 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.247727    2577 scope.go:117] "RemoveContainer" containerID="6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502"
Sep 18 20:31:41.254694 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.254676    2577 scope.go:117] "RemoveContainer" containerID="44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d"
Sep 18 20:31:41.254950 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:41.254929    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d\": container with ID starting with 44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d not found: ID does not exist" containerID="44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d"
Sep 18 20:31:41.255011 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.254958    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d"} err="failed to get container status \"44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d\": rpc error: code = NotFound desc = could not find container \"44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d\": container with ID starting with 44610b7f7ac2819bd69e4135ceff168afde3751873aab0055c1ce654ebcff76d not found: ID does not exist"
Sep 18 20:31:41.255011 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.254976    2577 scope.go:117] "RemoveContainer" containerID="c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e"
Sep 18 20:31:41.255210 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:41.255193    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e\": container with ID starting with c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e not found: ID does not exist" containerID="c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e"
Sep 18 20:31:41.255252 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.255215    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e"} err="failed to get container status \"c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e\": rpc error: code = NotFound desc = could not find container \"c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e\": container with ID starting with c7077987b70532ac199faf2ef8c1143393f44faa2b16d051c15b4897023eae0e not found: ID does not exist"
Sep 18 20:31:41.255252 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.255229    2577 scope.go:117] "RemoveContainer" containerID="346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901"
Sep 18 20:31:41.255469 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:41.255452    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901\": container with ID starting with 346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901 not found: ID does not exist" containerID="346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901"
Sep 18 20:31:41.255542 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.255486    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901"} err="failed to get container status \"346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901\": rpc error: code = NotFound desc = could not find container \"346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901\": container with ID starting with 346ede07f02c55de60782b881f816fd5cda7afee0672af12f412e0200c4f8901 not found: ID does not exist"
Sep 18 20:31:41.255542 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.255501    2577 scope.go:117] "RemoveContainer" containerID="6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502"
Sep 18 20:31:41.255721 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:31:41.255704    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502\": container with ID starting with 6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502 not found: ID does not exist" containerID="6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502"
Sep 18 20:31:41.255761 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.255724    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502"} err="failed to get container status \"6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502\": rpc error: code = NotFound desc = could not find container \"6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502\": container with ID starting with 6d3e9f060e458276f59567a64435ee71e5ec93c68956304e218e051069ed6502 not found: ID does not exist"
Sep 18 20:31:41.332260 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.332240    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:41.332361 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.332261    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:41.332361 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.332273    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0825e48c-e485-4b4b-9f7f-47b195eb5138-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:41.332361 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.332282    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6km9k\" (UniqueName: \"kubernetes.io/projected/0825e48c-e485-4b4b-9f7f-47b195eb5138-kube-api-access-6km9k\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:41.332361 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.332290    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0825e48c-e485-4b4b-9f7f-47b195eb5138-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:31:41.538289 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.538260    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"]
Sep 18 20:31:41.543542 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:41.543518    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d48f1-predictor-5ff47bbdcd-5hfws"]
Sep 18 20:31:43.207606 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:43.207580    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:31:43.208132 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:43.208104    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:31:43.242813 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:43.242783    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" path="/var/lib/kubelet/pods/0825e48c-e485-4b4b-9f7f-47b195eb5138/volumes"
Sep 18 20:31:49.101179 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:49.101138    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:31:53.208029 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:53.207986    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:31:59.101073 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:31:59.101031    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:32:03.208706 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:03.208664    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:32:09.101180 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:09.101141    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:32:13.208557 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:13.208510    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:32:19.100998 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:19.100955    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:32:23.208640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:23.208598    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:32:29.101618 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:29.101576    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 20:32:33.208775 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:33.208733    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:32:33.243301 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:33.243274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:32:35.768728 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:35.768690    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"]
Sep 18 20:32:35.769118 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:35.768937    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container" containerID="cri-o://8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266" gracePeriod=30
Sep 18 20:32:35.769118 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:35.768981    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kube-rbac-proxy" containerID="cri-o://9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749" gracePeriod=30
Sep 18 20:32:36.428322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:36.428288    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerID="9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749" exitCode=2
Sep 18 20:32:36.428524 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:36.428359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" event={"ID":"5bf6d415-5bf2-4d9b-9a94-0f36235d3140","Type":"ContainerDied","Data":"9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749"}
Sep 18 20:32:39.095015 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:39.094978    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 18 20:32:40.604702 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.604679    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:32:40.703324 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.703249    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rql9g\" (UniqueName: \"kubernetes.io/projected/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-api-access-rql9g\") pod \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") "
Sep 18 20:32:40.703324 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.703303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kserve-provision-location\") pod \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") "
Sep 18 20:32:40.703533 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.703346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-cabundle-cert\") pod \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") "
Sep 18 20:32:40.703533 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.703364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-rbac-proxy-sar-config\") pod \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") "
Sep 18 20:32:40.703533 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.703410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-proxy-tls\") pod \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\" (UID: \"5bf6d415-5bf2-4d9b-9a94-0f36235d3140\") "
Sep 18 20:32:40.703722 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.703699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bf6d415-5bf2-4d9b-9a94-0f36235d3140" (UID: "5bf6d415-5bf2-4d9b-9a94-0f36235d3140"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:40.703785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.703738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5bf6d415-5bf2-4d9b-9a94-0f36235d3140" (UID: "5bf6d415-5bf2-4d9b-9a94-0f36235d3140"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:40.703785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.703767    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bf6d415-5bf2-4d9b-9a94-0f36235d3140" (UID: "5bf6d415-5bf2-4d9b-9a94-0f36235d3140"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:40.705545 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.705512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bf6d415-5bf2-4d9b-9a94-0f36235d3140" (UID: "5bf6d415-5bf2-4d9b-9a94-0f36235d3140"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:32:40.705667 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.705569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-api-access-rql9g" (OuterVolumeSpecName: "kube-api-access-rql9g") pod "5bf6d415-5bf2-4d9b-9a94-0f36235d3140" (UID: "5bf6d415-5bf2-4d9b-9a94-0f36235d3140"). InnerVolumeSpecName "kube-api-access-rql9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:40.804361 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.804332    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.804361 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.804356    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.804361 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.804366    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.804570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.804374    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rql9g\" (UniqueName: \"kubernetes.io/projected/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kube-api-access-rql9g\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:32:40.804570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:40.804383    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf6d415-5bf2-4d9b-9a94-0f36235d3140-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:32:41.024319 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024241    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"]
Sep 18 20:32:41.024655 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024638    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container"
Sep 18 20:32:41.024655 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024657    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024680    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024688    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024701    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kube-rbac-proxy"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024708    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kube-rbac-proxy"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024718    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="storage-initializer"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024727    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="storage-initializer"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024742    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="storage-initializer"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024750    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="storage-initializer"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024758    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024766    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024780    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container"
Sep 18 20:32:41.024812 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024788    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container"
Sep 18 20:32:41.025415 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024879    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kserve-container"
Sep 18 20:32:41.025415 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024891    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kube-rbac-proxy"
Sep 18 20:32:41.025415 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024898    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerName="kube-rbac-proxy"
Sep 18 20:32:41.025415 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024911    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="kserve-container"
Sep 18 20:32:41.025415 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.024919    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0825e48c-e485-4b4b-9f7f-47b195eb5138" containerName="agent"
Sep 18 20:32:41.028415 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.028392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.032085 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.032066    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 18 20:32:41.033088 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.033071    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 20:32:41.042097 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.042076    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"]
Sep 18 20:32:41.106714 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.106687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.106863 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.106726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.106863 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.106833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.106974 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.106868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhpj\" (UniqueName: \"kubernetes.io/projected/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-api-access-7zhpj\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.106974 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.106907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.127803 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.127774    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"]
Sep 18 20:32:41.131054 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.131038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.133847 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.133828    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 20:32:41.144204 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.144186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"]
Sep 18 20:32:41.207755 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.207724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af964572-fc1b-4917-aff8-5e95f8736d67-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.207929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.207771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.207929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.207821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhpj\" (UniqueName: \"kubernetes.io/projected/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-api-access-7zhpj\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.207929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.207856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.207929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.207884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6vd\" (UniqueName: \"kubernetes.io/projected/af964572-fc1b-4917-aff8-5e95f8736d67-kube-api-access-ql6vd\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.208127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.207963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.208127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.207990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.208127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.208024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.208127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.208047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.208297 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.208126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af964572-fc1b-4917-aff8-5e95f8736d67-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.208297 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.208146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.208747 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.208727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.208805 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.208736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.210234 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.210215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.215720 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.215703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhpj\" (UniqueName: \"kubernetes.io/projected/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-api-access-7zhpj\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.309432 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.309374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6vd\" (UniqueName: \"kubernetes.io/projected/af964572-fc1b-4917-aff8-5e95f8736d67-kube-api-access-ql6vd\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.309432 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.309429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.309630 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.309446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.309630 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.309471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af964572-fc1b-4917-aff8-5e95f8736d67-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.309630 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.309524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af964572-fc1b-4917-aff8-5e95f8736d67-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.309921 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.309900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af964572-fc1b-4917-aff8-5e95f8736d67-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.310107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.310089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.310161 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.310129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.311731 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.311715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af964572-fc1b-4917-aff8-5e95f8736d67-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.317564 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.317542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6vd\" (UniqueName: \"kubernetes.io/projected/af964572-fc1b-4917-aff8-5e95f8736d67-kube-api-access-ql6vd\") pod \"isvc-canary-stop-predictor-5ffdccb94c-bj772\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.338922 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.338902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:41.441104 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.441076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:41.448959 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.448929    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" containerID="8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266" exitCode=0
Sep 18 20:32:41.449048 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.448993    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"
Sep 18 20:32:41.449048 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.449013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" event={"ID":"5bf6d415-5bf2-4d9b-9a94-0f36235d3140","Type":"ContainerDied","Data":"8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266"}
Sep 18 20:32:41.449120 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.449078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r" event={"ID":"5bf6d415-5bf2-4d9b-9a94-0f36235d3140","Type":"ContainerDied","Data":"242b79e04c41b710f261c3811aa9d401fa6e514a689cf44db7f5b601b16e26bd"}
Sep 18 20:32:41.449120 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.449109    2577 scope.go:117] "RemoveContainer" containerID="9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749"
Sep 18 20:32:41.466641 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.466620    2577 scope.go:117] "RemoveContainer" containerID="8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266"
Sep 18 20:32:41.473842 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.473824    2577 scope.go:117] "RemoveContainer" containerID="0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe"
Sep 18 20:32:41.476547 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.476526    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"]
Sep 18 20:32:41.483377 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.483296    2577 scope.go:117] "RemoveContainer" containerID="9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749"
Sep 18 20:32:41.483714 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:32:41.483682    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749\": container with ID starting with 9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749 not found: ID does not exist" containerID="9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749"
Sep 18 20:32:41.483800 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.483717    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749"} err="failed to get container status \"9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749\": rpc error: code = NotFound desc = could not find container \"9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749\": container with ID starting with 9d58f0b5d12fa74766a894c212d5af2d415cefab461a346e3d31a37582aec749 not found: ID does not exist"
Sep 18 20:32:41.483800 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.483754    2577 scope.go:117] "RemoveContainer" containerID="8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266"
Sep 18 20:32:41.484108 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:32:41.484080    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266\": container with ID starting with 8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266 not found: ID does not exist" containerID="8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266"
Sep 18 20:32:41.484159 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.484108    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266"} err="failed to get container status \"8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266\": rpc error: code = NotFound desc = could not find container \"8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266\": container with ID starting with 8594789285cd16046ab1d0ef620d9db2f49c9c67ba29c555b8ab67964b0db266 not found: ID does not exist"
Sep 18 20:32:41.484159 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.484125    2577 scope.go:117] "RemoveContainer" containerID="0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe"
Sep 18 20:32:41.484405 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:32:41.484387    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe\": container with ID starting with 0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe not found: ID does not exist" containerID="0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe"
Sep 18 20:32:41.484488 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.484408    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe"} err="failed to get container status \"0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe\": rpc error: code = NotFound desc = could not find container \"0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe\": container with ID starting with 0f0b52db0d309e73e213f96089a7e7016631d25e2e82d936d7a41c90c5b677fe not found: ID does not exist"
Sep 18 20:32:41.484842 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.484827    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-89s7r"]
Sep 18 20:32:41.607363 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.607257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"]
Sep 18 20:32:41.610115 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:32:41.610081    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbf9678_9ddf_4056_a6fc_0b8714a5b5c0.slice/crio-6d3b8b94717ca5fa3925808d5d01edf44a0476c5c298ef61c9352f4d8234fee9 WatchSource:0}: Error finding container 6d3b8b94717ca5fa3925808d5d01edf44a0476c5c298ef61c9352f4d8234fee9: Status 404 returned error can't find the container with id 6d3b8b94717ca5fa3925808d5d01edf44a0476c5c298ef61c9352f4d8234fee9
Sep 18 20:32:41.713646 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:41.713625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"]
Sep 18 20:32:41.715482 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:32:41.715442    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf964572_fc1b_4917_aff8_5e95f8736d67.slice/crio-4660b0f2d65d1a4ddde176dda9e4351b6e2dd5876d36f55ce9f530dc5282b4e0 WatchSource:0}: Error finding container 4660b0f2d65d1a4ddde176dda9e4351b6e2dd5876d36f55ce9f530dc5282b4e0: Status 404 returned error can't find the container with id 4660b0f2d65d1a4ddde176dda9e4351b6e2dd5876d36f55ce9f530dc5282b4e0
Sep 18 20:32:42.455673 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:42.455636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" event={"ID":"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0","Type":"ContainerStarted","Data":"bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34"}
Sep 18 20:32:42.455833 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:42.455702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" event={"ID":"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0","Type":"ContainerStarted","Data":"6d3b8b94717ca5fa3925808d5d01edf44a0476c5c298ef61c9352f4d8234fee9"}
Sep 18 20:32:42.457077 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:42.457052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" event={"ID":"af964572-fc1b-4917-aff8-5e95f8736d67","Type":"ContainerStarted","Data":"42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a"}
Sep 18 20:32:42.457187 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:42.457108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" event={"ID":"af964572-fc1b-4917-aff8-5e95f8736d67","Type":"ContainerStarted","Data":"4660b0f2d65d1a4ddde176dda9e4351b6e2dd5876d36f55ce9f530dc5282b4e0"}
Sep 18 20:32:43.208637 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:43.208605    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:32:43.244008 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:43.243977    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf6d415-5bf2-4d9b-9a94-0f36235d3140" path="/var/lib/kubelet/pods/5bf6d415-5bf2-4d9b-9a94-0f36235d3140/volumes"
Sep 18 20:32:46.474122 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:46.474088    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerID="bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34" exitCode=0
Sep 18 20:32:46.474547 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:46.474161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" event={"ID":"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0","Type":"ContainerDied","Data":"bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34"}
Sep 18 20:32:46.475662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:46.475637    2577 generic.go:356] "Generic (PLEG): container finished" podID="af964572-fc1b-4917-aff8-5e95f8736d67" containerID="42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a" exitCode=0
Sep 18 20:32:46.475749 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:46.475710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" event={"ID":"af964572-fc1b-4917-aff8-5e95f8736d67","Type":"ContainerDied","Data":"42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a"}
Sep 18 20:32:47.481500 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.481440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" event={"ID":"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0","Type":"ContainerStarted","Data":"930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a"}
Sep 18 20:32:47.481962 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.481531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" event={"ID":"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0","Type":"ContainerStarted","Data":"f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578"}
Sep 18 20:32:47.481962 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.481828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:47.482079 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.481984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:47.483579 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.483553    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:32:47.483781 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.483761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" event={"ID":"af964572-fc1b-4917-aff8-5e95f8736d67","Type":"ContainerStarted","Data":"4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4"}
Sep 18 20:32:47.483844 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.483804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" event={"ID":"af964572-fc1b-4917-aff8-5e95f8736d67","Type":"ContainerStarted","Data":"2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f"}
Sep 18 20:32:47.484060 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.484046    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:47.484116 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.484087    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:47.485132 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.485107    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:32:47.501522 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.501464    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podStartSLOduration=2.623650812 podStartE2EDuration="6.501452935s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877802123s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:47.500257936 +0000 UTC m=+744.310043878" watchObservedRunningTime="2026-09-18 20:32:47.501452935 +0000 UTC m=+744.311238919"
Sep 18 20:32:47.519805 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:47.519767    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podStartSLOduration=2.66719346 podStartE2EDuration="6.519750738s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.852557278s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:47.518589429 +0000 UTC m=+744.328375398" watchObservedRunningTime="2026-09-18 20:32:47.519750738 +0000 UTC m=+744.329536680"
Sep 18 20:32:48.487638 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:48.487598    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:32:48.488040 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:48.487681    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:32:51.245899 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.245864    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"]
Sep 18 20:32:51.249389 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.249373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:51.252694 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.252667    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-12fdc-serving-cert\""
Sep 18 20:32:51.252694 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.252669    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-12fdc-kube-rbac-proxy-sar-config\""
Sep 18 20:32:51.260000 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.259975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"]
Sep 18 20:32:51.289180 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.289151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-openshift-service-ca-bundle\") pod \"model-chainer-raw-12fdc-b7d685587-n7pvl\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") " pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:51.289319 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.289194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls\") pod \"model-chainer-raw-12fdc-b7d685587-n7pvl\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") " pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:51.390163 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.390132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls\") pod \"model-chainer-raw-12fdc-b7d685587-n7pvl\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") " pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:51.390306 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:32:51.390281    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-12fdc-serving-cert: secret "model-chainer-raw-12fdc-serving-cert" not found
Sep 18 20:32:51.390384 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.390305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-openshift-service-ca-bundle\") pod \"model-chainer-raw-12fdc-b7d685587-n7pvl\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") " pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:51.390384 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:32:51.390343    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls podName:b53ac6a4-55b4-4347-b75c-fae1ec83e52b nodeName:}" failed. No retries permitted until 2026-09-18 20:32:51.890326267 +0000 UTC m=+748.700112186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls") pod "model-chainer-raw-12fdc-b7d685587-n7pvl" (UID: "b53ac6a4-55b4-4347-b75c-fae1ec83e52b") : secret "model-chainer-raw-12fdc-serving-cert" not found
Sep 18 20:32:51.390932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.390909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-openshift-service-ca-bundle\") pod \"model-chainer-raw-12fdc-b7d685587-n7pvl\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") " pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:51.895373 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.895329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls\") pod \"model-chainer-raw-12fdc-b7d685587-n7pvl\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") " pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:51.897796 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:51.897772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls\") pod \"model-chainer-raw-12fdc-b7d685587-n7pvl\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") " pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:52.158892 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:52.158805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:52.426762 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:52.426739    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"]
Sep 18 20:32:52.428884 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:32:52.428856    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53ac6a4_55b4_4347_b75c_fae1ec83e52b.slice/crio-a7a7bc1da2d4cf4107d0da614e63d11cd74f1e12a06bd0a43491d03032342c8b WatchSource:0}: Error finding container a7a7bc1da2d4cf4107d0da614e63d11cd74f1e12a06bd0a43491d03032342c8b: Status 404 returned error can't find the container with id a7a7bc1da2d4cf4107d0da614e63d11cd74f1e12a06bd0a43491d03032342c8b
Sep 18 20:32:52.505434 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:52.505405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" event={"ID":"b53ac6a4-55b4-4347-b75c-fae1ec83e52b","Type":"ContainerStarted","Data":"a7a7bc1da2d4cf4107d0da614e63d11cd74f1e12a06bd0a43491d03032342c8b"}
Sep 18 20:32:53.493559 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:53.493531    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:32:53.493919 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:53.493772    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:32:53.494086 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:53.494056    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:32:53.494499 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:53.494447    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:32:54.514666 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:54.514619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" event={"ID":"b53ac6a4-55b4-4347-b75c-fae1ec83e52b","Type":"ContainerStarted","Data":"d11ecedae35649fd38be886470a2994977625951ee629a016c113d5c8b5031bb"}
Sep 18 20:32:54.515037 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:54.514875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:32:54.533928 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:32:54.533889    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" podStartSLOduration=1.628716807 podStartE2EDuration="3.533875255s" totalImagesPullingTime="1.905158448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:54.532362156 +0000 UTC m=+751.342148098" watchObservedRunningTime="2026-09-18 20:32:54.533875255 +0000 UTC m=+751.343661214"
Sep 18 20:33:00.524912 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:00.524884    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:33:01.272751 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.272721    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"]
Sep 18 20:33:01.272922 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.272885    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc" containerID="cri-o://d11ecedae35649fd38be886470a2994977625951ee629a016c113d5c8b5031bb" gracePeriod=30
Sep 18 20:33:01.431578 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.431537    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"]
Sep 18 20:33:01.431828 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.431785    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" containerID="cri-o://7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919" gracePeriod=30
Sep 18 20:33:01.431923 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.431900    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kube-rbac-proxy" containerID="cri-o://6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109" gracePeriod=30
Sep 18 20:33:01.547419 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.547342    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"]
Sep 18 20:33:01.551168 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.551149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.553461 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.553442    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e4001-predictor-serving-cert\""
Sep 18 20:33:01.553592 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.553556    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e4001-kube-rbac-proxy-sar-config\""
Sep 18 20:33:01.561736 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.561714    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"]
Sep 18 20:33:01.658625 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.658585    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"]
Sep 18 20:33:01.662718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.662696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.664999 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.664976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e4001-predictor-serving-cert\""
Sep 18 20:33:01.665127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.664978    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e4001-kube-rbac-proxy-sar-config\""
Sep 18 20:33:01.675649 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.675629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"]
Sep 18 20:33:01.683004 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.682983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjlf\" (UniqueName: \"kubernetes.io/projected/ff123e27-496f-4b9e-8c81-834504e16726-kube-api-access-jvjlf\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.683127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.683011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.683127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.683031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.683127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.683066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.683127 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.683111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff123e27-496f-4b9e-8c81-834504e16726-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.784416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjlf\" (UniqueName: \"kubernetes.io/projected/ff123e27-496f-4b9e-8c81-834504e16726-kube-api-access-jvjlf\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.784416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.784662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.784662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.784662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.784662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.784662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff123e27-496f-4b9e-8c81-834504e16726-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.784662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxgl\" (UniqueName: \"kubernetes.io/projected/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-api-access-flxgl\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.784951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.784951 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:33:01.784685    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-e4001-predictor-serving-cert" not found
Sep 18 20:33:01.784951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.784733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64527dec-25cb-4226-b2bb-f10738bb36ce-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.784951 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:33:01.784759    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls podName:ff123e27-496f-4b9e-8c81-834504e16726 nodeName:}" failed. No retries permitted until 2026-09-18 20:33:02.284736737 +0000 UTC m=+759.094522664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" (UID: "ff123e27-496f-4b9e-8c81-834504e16726") : secret "isvc-sklearn-graph-raw-hpa-e4001-predictor-serving-cert" not found
Sep 18 20:33:01.785166 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.785027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff123e27-496f-4b9e-8c81-834504e16726-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.785248 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.785230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.785331 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.785307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.792955 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.792929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjlf\" (UniqueName: \"kubernetes.io/projected/ff123e27-496f-4b9e-8c81-834504e16726-kube-api-access-jvjlf\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:01.885605 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.885568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64527dec-25cb-4226-b2bb-f10738bb36ce-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.885758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.885666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.885758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.885694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.885758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.885737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxgl\" (UniqueName: \"kubernetes.io/projected/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-api-access-flxgl\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.885980 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.885782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.885980 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:33:01.885901    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-e4001-predictor-serving-cert" not found
Sep 18 20:33:01.885980 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:33:01.885960    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls podName:64527dec-25cb-4226-b2bb-f10738bb36ce nodeName:}" failed. No retries permitted until 2026-09-18 20:33:02.385940562 +0000 UTC m=+759.195726481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" (UID: "64527dec-25cb-4226-b2bb-f10738bb36ce") : secret "isvc-xgboost-graph-raw-hpa-e4001-predictor-serving-cert" not found
Sep 18 20:33:01.886149 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.886035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64527dec-25cb-4226-b2bb-f10738bb36ce-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.886524 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.886492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.886739 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.886723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:01.895969 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:01.895939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxgl\" (UniqueName: \"kubernetes.io/projected/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-api-access-flxgl\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:02.287915 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.287826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:02.290158 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.290129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:02.388432 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.388396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:02.390833 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.390807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:02.461379 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.461348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:02.549727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.549655    2577 generic.go:356] "Generic (PLEG): container finished" podID="f17e442f-ac76-493b-a93f-18334ff6354a" containerID="6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109" exitCode=2
Sep 18 20:33:02.550024 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.549721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" event={"ID":"f17e442f-ac76-493b-a93f-18334ff6354a","Type":"ContainerDied","Data":"6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109"}
Sep 18 20:33:02.573941 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.573923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:02.734880 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.734854    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"]
Sep 18 20:33:02.737166 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:33:02.737135    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff123e27_496f_4b9e_8c81_834504e16726.slice/crio-7fa908d7c85ce6a67cd6699c35643262304cf3e7262dcb57dca3d6d6cee436ca WatchSource:0}: Error finding container 7fa908d7c85ce6a67cd6699c35643262304cf3e7262dcb57dca3d6d6cee436ca: Status 404 returned error can't find the container with id 7fa908d7c85ce6a67cd6699c35643262304cf3e7262dcb57dca3d6d6cee436ca
Sep 18 20:33:02.847786 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:02.847761    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"]
Sep 18 20:33:02.850664 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:33:02.850636    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64527dec_25cb_4226_b2bb_f10738bb36ce.slice/crio-4d1c0bc74a3022d729e312f31d1b15fb9f26e23d4b2f010df27af38c97f6606e WatchSource:0}: Error finding container 4d1c0bc74a3022d729e312f31d1b15fb9f26e23d4b2f010df27af38c97f6606e: Status 404 returned error can't find the container with id 4d1c0bc74a3022d729e312f31d1b15fb9f26e23d4b2f010df27af38c97f6606e
Sep 18 20:33:03.202091 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:03.201996    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 18 20:33:03.208676 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:03.208641    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 20:33:03.495044 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:03.494970    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:33:03.495172 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:03.494972    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:33:03.560681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:03.560644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" event={"ID":"64527dec-25cb-4226-b2bb-f10738bb36ce","Type":"ContainerStarted","Data":"cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442"}
Sep 18 20:33:03.561091 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:03.560761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" event={"ID":"64527dec-25cb-4226-b2bb-f10738bb36ce","Type":"ContainerStarted","Data":"4d1c0bc74a3022d729e312f31d1b15fb9f26e23d4b2f010df27af38c97f6606e"}
Sep 18 20:33:03.562186 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:03.562155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" event={"ID":"ff123e27-496f-4b9e-8c81-834504e16726","Type":"ContainerStarted","Data":"0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56"}
Sep 18 20:33:03.562301 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:03.562210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" event={"ID":"ff123e27-496f-4b9e-8c81-834504e16726","Type":"ContainerStarted","Data":"7fa908d7c85ce6a67cd6699c35643262304cf3e7262dcb57dca3d6d6cee436ca"}
Sep 18 20:33:05.479936 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.479913    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:33:05.524352 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.524313    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:05.571002 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.570931    2577 generic.go:356] "Generic (PLEG): container finished" podID="f17e442f-ac76-493b-a93f-18334ff6354a" containerID="7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919" exitCode=0
Sep 18 20:33:05.571002 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.570992    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"
Sep 18 20:33:05.571192 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.571012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" event={"ID":"f17e442f-ac76-493b-a93f-18334ff6354a","Type":"ContainerDied","Data":"7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919"}
Sep 18 20:33:05.571192 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.571090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244" event={"ID":"f17e442f-ac76-493b-a93f-18334ff6354a","Type":"ContainerDied","Data":"0dd8196b174d1366a83b860f4572a5637e14cd9aece1626473bae1db438ac5e7"}
Sep 18 20:33:05.571192 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.571130    2577 scope.go:117] "RemoveContainer" containerID="6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109"
Sep 18 20:33:05.578966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.578952    2577 scope.go:117] "RemoveContainer" containerID="7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919"
Sep 18 20:33:05.586172 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.586155    2577 scope.go:117] "RemoveContainer" containerID="e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff"
Sep 18 20:33:05.593086 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.593069    2577 scope.go:117] "RemoveContainer" containerID="6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109"
Sep 18 20:33:05.593362 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:33:05.593341    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109\": container with ID starting with 6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109 not found: ID does not exist" containerID="6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109"
Sep 18 20:33:05.593404 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.593381    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109"} err="failed to get container status \"6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109\": rpc error: code = NotFound desc = could not find container \"6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109\": container with ID starting with 6fefd7ebcda0c725f1fbd9dd31e8750460687e3a46b723d2f58ff8bdf0400109 not found: ID does not exist"
Sep 18 20:33:05.593404 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.593396    2577 scope.go:117] "RemoveContainer" containerID="7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919"
Sep 18 20:33:05.593657 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:33:05.593637    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919\": container with ID starting with 7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919 not found: ID does not exist" containerID="7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919"
Sep 18 20:33:05.593757 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.593663    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919"} err="failed to get container status \"7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919\": rpc error: code = NotFound desc = could not find container \"7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919\": container with ID starting with 7c237d8eb3006577d6db416578381f98deacaacc82de2d068513f534346ce919 not found: ID does not exist"
Sep 18 20:33:05.593757 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.593680    2577 scope.go:117] "RemoveContainer" containerID="e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff"
Sep 18 20:33:05.593965 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:33:05.593949    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff\": container with ID starting with e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff not found: ID does not exist" containerID="e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff"
Sep 18 20:33:05.594009 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.593970    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff"} err="failed to get container status \"e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff\": rpc error: code = NotFound desc = could not find container \"e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff\": container with ID starting with e474211d7086c77eeac405eba857d6e6d9b7b110c5490b8c98916b531781d5ff not found: ID does not exist"
Sep 18 20:33:05.616908 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.616886    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-kube-rbac-proxy-sar-config\") pod \"f17e442f-ac76-493b-a93f-18334ff6354a\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") "
Sep 18 20:33:05.616991 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.616954    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f17e442f-ac76-493b-a93f-18334ff6354a-kserve-provision-location\") pod \"f17e442f-ac76-493b-a93f-18334ff6354a\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") "
Sep 18 20:33:05.617141 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.617122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh6qq\" (UniqueName: \"kubernetes.io/projected/f17e442f-ac76-493b-a93f-18334ff6354a-kube-api-access-vh6qq\") pod \"f17e442f-ac76-493b-a93f-18334ff6354a\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") "
Sep 18 20:33:05.617215 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.617172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-cabundle-cert\") pod \"f17e442f-ac76-493b-a93f-18334ff6354a\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") "
Sep 18 20:33:05.617266 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.617226    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f17e442f-ac76-493b-a93f-18334ff6354a-proxy-tls\") pod \"f17e442f-ac76-493b-a93f-18334ff6354a\" (UID: \"f17e442f-ac76-493b-a93f-18334ff6354a\") "
Sep 18 20:33:05.617266 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.617230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17e442f-ac76-493b-a93f-18334ff6354a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f17e442f-ac76-493b-a93f-18334ff6354a" (UID: "f17e442f-ac76-493b-a93f-18334ff6354a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:05.617266 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.617250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f17e442f-ac76-493b-a93f-18334ff6354a" (UID: "f17e442f-ac76-493b-a93f-18334ff6354a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:05.617505 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.617468    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f17e442f-ac76-493b-a93f-18334ff6354a-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:33:05.617505 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.617506    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:33:05.617660 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.617544    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f17e442f-ac76-493b-a93f-18334ff6354a" (UID: "f17e442f-ac76-493b-a93f-18334ff6354a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:05.619661 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.619639    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17e442f-ac76-493b-a93f-18334ff6354a-kube-api-access-vh6qq" (OuterVolumeSpecName: "kube-api-access-vh6qq") pod "f17e442f-ac76-493b-a93f-18334ff6354a" (UID: "f17e442f-ac76-493b-a93f-18334ff6354a"). InnerVolumeSpecName "kube-api-access-vh6qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:33:05.619661 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.619646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17e442f-ac76-493b-a93f-18334ff6354a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f17e442f-ac76-493b-a93f-18334ff6354a" (UID: "f17e442f-ac76-493b-a93f-18334ff6354a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:33:05.718775 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.718750    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh6qq\" (UniqueName: \"kubernetes.io/projected/f17e442f-ac76-493b-a93f-18334ff6354a-kube-api-access-vh6qq\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:33:05.718775 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.718772    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f17e442f-ac76-493b-a93f-18334ff6354a-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:33:05.718905 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.718782    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f17e442f-ac76-493b-a93f-18334ff6354a-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:33:05.893591 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.893564    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"]
Sep 18 20:33:05.897540 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:05.897519    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-12fdc-predictor-54d7d54b5-8b244"]
Sep 18 20:33:07.243402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:07.243360    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" path="/var/lib/kubelet/pods/f17e442f-ac76-493b-a93f-18334ff6354a/volumes"
Sep 18 20:33:07.581698 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:07.581667    2577 generic.go:356] "Generic (PLEG): container finished" podID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerID="cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442" exitCode=0
Sep 18 20:33:07.581878 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:07.581746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" event={"ID":"64527dec-25cb-4226-b2bb-f10738bb36ce","Type":"ContainerDied","Data":"cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442"}
Sep 18 20:33:07.583350 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:07.583329    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff123e27-496f-4b9e-8c81-834504e16726" containerID="0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56" exitCode=0
Sep 18 20:33:07.583435 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:07.583410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" event={"ID":"ff123e27-496f-4b9e-8c81-834504e16726","Type":"ContainerDied","Data":"0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56"}
Sep 18 20:33:08.590056 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.590023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" event={"ID":"64527dec-25cb-4226-b2bb-f10738bb36ce","Type":"ContainerStarted","Data":"a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d"}
Sep 18 20:33:08.590457 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.590090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" event={"ID":"64527dec-25cb-4226-b2bb-f10738bb36ce","Type":"ContainerStarted","Data":"ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642"}
Sep 18 20:33:08.590457 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.590282    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:08.592227 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.592205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" event={"ID":"ff123e27-496f-4b9e-8c81-834504e16726","Type":"ContainerStarted","Data":"9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34"}
Sep 18 20:33:08.592323 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.592250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" event={"ID":"ff123e27-496f-4b9e-8c81-834504e16726","Type":"ContainerStarted","Data":"3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7"}
Sep 18 20:33:08.592506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.592491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:08.592558 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.592529    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:08.593768 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.593747    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:33:08.614774 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.614730    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podStartSLOduration=3.893202355 podStartE2EDuration="7.614719301s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721516946s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:33:08.612731574 +0000 UTC m=+765.422517531" watchObservedRunningTime="2026-09-18 20:33:08.614719301 +0000 UTC m=+765.424505242"
Sep 18 20:33:08.638132 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:08.638089    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podStartSLOduration=3.767973653 podStartE2EDuration="7.638074192s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.870100539s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:33:08.636867256 +0000 UTC m=+765.446653198" watchObservedRunningTime="2026-09-18 20:33:08.638074192 +0000 UTC m=+765.447860133"
Sep 18 20:33:09.596866 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:09.596826    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:33:09.597321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:09.597103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:09.598228 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:09.598203    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:10.524629 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:10.524596    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:10.601047 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:10.601012    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:13.494291 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:13.494255    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:33:13.494690 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:13.494255    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:33:14.600913 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:14.600885    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:33:14.601611 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:14.601581    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:33:15.524590 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:15.524551    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:15.524769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:15.524672    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:33:15.605793 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:15.605766    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:33:15.606384 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:15.606293    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:20.524389 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:20.524352    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:23.494433 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:23.494397    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:33:23.494895 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:23.494397    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:33:24.602288 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:24.602244    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:33:25.523969 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:25.523870    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:25.606403 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:25.606363    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:30.524987 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:30.524938    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:31.693281 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:31.693243    2577 generic.go:356] "Generic (PLEG): container finished" podID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerID="d11ecedae35649fd38be886470a2994977625951ee629a016c113d5c8b5031bb" exitCode=0
Sep 18 20:33:31.693671 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:31.693318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" event={"ID":"b53ac6a4-55b4-4347-b75c-fae1ec83e52b","Type":"ContainerDied","Data":"d11ecedae35649fd38be886470a2994977625951ee629a016c113d5c8b5031bb"}
Sep 18 20:33:31.914815 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:31.914783    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:33:31.943866 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:31.943779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls\") pod \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") "
Sep 18 20:33:31.944017 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:31.943873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-openshift-service-ca-bundle\") pod \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\" (UID: \"b53ac6a4-55b4-4347-b75c-fae1ec83e52b\") "
Sep 18 20:33:31.944211 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:31.944184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b53ac6a4-55b4-4347-b75c-fae1ec83e52b" (UID: "b53ac6a4-55b4-4347-b75c-fae1ec83e52b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:31.945830 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:31.945810    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b53ac6a4-55b4-4347-b75c-fae1ec83e52b" (UID: "b53ac6a4-55b4-4347-b75c-fae1ec83e52b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:33:32.045195 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:32.045157    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:33:32.045195 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:32.045187    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b53ac6a4-55b4-4347-b75c-fae1ec83e52b-openshift-service-ca-bundle\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:33:32.699471 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:32.699445    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"
Sep 18 20:33:32.699855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:32.699447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl" event={"ID":"b53ac6a4-55b4-4347-b75c-fae1ec83e52b","Type":"ContainerDied","Data":"a7a7bc1da2d4cf4107d0da614e63d11cd74f1e12a06bd0a43491d03032342c8b"}
Sep 18 20:33:32.699855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:32.699603    2577 scope.go:117] "RemoveContainer" containerID="d11ecedae35649fd38be886470a2994977625951ee629a016c113d5c8b5031bb"
Sep 18 20:33:32.720257 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:32.720232    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"]
Sep 18 20:33:32.723280 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:32.723255    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-12fdc-b7d685587-n7pvl"]
Sep 18 20:33:33.242971 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:33.242937    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" path="/var/lib/kubelet/pods/b53ac6a4-55b4-4347-b75c-fae1ec83e52b/volumes"
Sep 18 20:33:33.495082 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:33.494997    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:33:33.495238 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:33.494997    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:33:34.601658 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:34.601618    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:33:35.606256 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:35.606219    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:43.494305 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:43.494224    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:33:43.494751 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:43.494365    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:33:44.602154 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:44.602116    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:33:45.606664 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:45.606617    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:53.494322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:53.494275    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:33:53.494731 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:53.494275    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:33:54.601862 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:54.601824    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:33:55.606356 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:55.606320    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:59.238774 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:33:59.238727    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 20:34:03.494197 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:03.494161    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:34:04.602170 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:04.602132    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:34:05.606677 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:05.606643    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:34:09.243817 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:09.243787    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:34:13.494627 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:13.494594    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:34:13.626601 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:13.626572    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"]
Sep 18 20:34:13.626795 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:13.626773    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container" containerID="cri-o://2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f" gracePeriod=30
Sep 18 20:34:13.626865 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:13.626833    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kube-rbac-proxy" containerID="cri-o://4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4" gracePeriod=30
Sep 18 20:34:13.879742 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:13.879713    2577 generic.go:356] "Generic (PLEG): container finished" podID="af964572-fc1b-4917-aff8-5e95f8736d67" containerID="4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4" exitCode=2
Sep 18 20:34:13.879910 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:13.879780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" event={"ID":"af964572-fc1b-4917-aff8-5e95f8736d67","Type":"ContainerDied","Data":"4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4"}
Sep 18 20:34:14.601630 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:14.601586    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:34:15.607201 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:15.607164    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:34:15.969030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:15.968957    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"]
Sep 18 20:34:15.969217 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:15.969193    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container" containerID="cri-o://f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578" gracePeriod=30
Sep 18 20:34:15.969273 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:15.969213    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kube-rbac-proxy" containerID="cri-o://930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a" gracePeriod=30
Sep 18 20:34:16.894983 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:16.894943    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerID="930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a" exitCode=2
Sep 18 20:34:16.895347 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:16.895005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" event={"ID":"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0","Type":"ContainerDied","Data":"930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a"}
Sep 18 20:34:18.488825 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.488790    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 18 20:34:18.489184 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.488790    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 18 20:34:18.668648 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.668622    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:34:18.834886 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.834857    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af964572-fc1b-4917-aff8-5e95f8736d67-kserve-provision-location\") pod \"af964572-fc1b-4917-aff8-5e95f8736d67\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") "
Sep 18 20:34:18.835044 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.834904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-cabundle-cert\") pod \"af964572-fc1b-4917-aff8-5e95f8736d67\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") "
Sep 18 20:34:18.835044 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.834951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af964572-fc1b-4917-aff8-5e95f8736d67-proxy-tls\") pod \"af964572-fc1b-4917-aff8-5e95f8736d67\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") "
Sep 18 20:34:18.835044 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.834977    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql6vd\" (UniqueName: \"kubernetes.io/projected/af964572-fc1b-4917-aff8-5e95f8736d67-kube-api-access-ql6vd\") pod \"af964572-fc1b-4917-aff8-5e95f8736d67\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") "
Sep 18 20:34:18.835044 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.835013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-kube-rbac-proxy-sar-config\") pod \"af964572-fc1b-4917-aff8-5e95f8736d67\" (UID: \"af964572-fc1b-4917-aff8-5e95f8736d67\") "
Sep 18 20:34:18.835241 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.835219    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af964572-fc1b-4917-aff8-5e95f8736d67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af964572-fc1b-4917-aff8-5e95f8736d67" (UID: "af964572-fc1b-4917-aff8-5e95f8736d67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:34:18.835383 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.835355    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af964572-fc1b-4917-aff8-5e95f8736d67" (UID: "af964572-fc1b-4917-aff8-5e95f8736d67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:34:18.835531 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.835398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af964572-fc1b-4917-aff8-5e95f8736d67" (UID: "af964572-fc1b-4917-aff8-5e95f8736d67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:34:18.837119 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.837091    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af964572-fc1b-4917-aff8-5e95f8736d67-kube-api-access-ql6vd" (OuterVolumeSpecName: "kube-api-access-ql6vd") pod "af964572-fc1b-4917-aff8-5e95f8736d67" (UID: "af964572-fc1b-4917-aff8-5e95f8736d67"). InnerVolumeSpecName "kube-api-access-ql6vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:34:18.837119 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.837096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af964572-fc1b-4917-aff8-5e95f8736d67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af964572-fc1b-4917-aff8-5e95f8736d67" (UID: "af964572-fc1b-4917-aff8-5e95f8736d67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:34:18.904637 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.904605    2577 generic.go:356] "Generic (PLEG): container finished" podID="af964572-fc1b-4917-aff8-5e95f8736d67" containerID="2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f" exitCode=0
Sep 18 20:34:18.904787 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.904641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" event={"ID":"af964572-fc1b-4917-aff8-5e95f8736d67","Type":"ContainerDied","Data":"2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f"}
Sep 18 20:34:18.904787 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.904676    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"
Sep 18 20:34:18.904787 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.904689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772" event={"ID":"af964572-fc1b-4917-aff8-5e95f8736d67","Type":"ContainerDied","Data":"4660b0f2d65d1a4ddde176dda9e4351b6e2dd5876d36f55ce9f530dc5282b4e0"}
Sep 18 20:34:18.904787 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.904715    2577 scope.go:117] "RemoveContainer" containerID="4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4"
Sep 18 20:34:18.913930 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.913912    2577 scope.go:117] "RemoveContainer" containerID="2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f"
Sep 18 20:34:18.925695 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.925676    2577 scope.go:117] "RemoveContainer" containerID="42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a"
Sep 18 20:34:18.927883 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.927865    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"]
Sep 18 20:34:18.931948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.931924    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-bj772"]
Sep 18 20:34:18.933709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.933692    2577 scope.go:117] "RemoveContainer" containerID="4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4"
Sep 18 20:34:18.933968 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:34:18.933951    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4\": container with ID starting with 4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4 not found: ID does not exist" containerID="4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4"
Sep 18 20:34:18.934010 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.933977    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4"} err="failed to get container status \"4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4\": rpc error: code = NotFound desc = could not find container \"4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4\": container with ID starting with 4368bcec90a8ba2cdc0b23ced7e7decc940ff79d14e504b90ecbdbeeabf9c7c4 not found: ID does not exist"
Sep 18 20:34:18.934010 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.933996    2577 scope.go:117] "RemoveContainer" containerID="2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f"
Sep 18 20:34:18.934265 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:34:18.934248    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f\": container with ID starting with 2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f not found: ID does not exist" containerID="2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f"
Sep 18 20:34:18.934310 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.934271    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f"} err="failed to get container status \"2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f\": rpc error: code = NotFound desc = could not find container \"2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f\": container with ID starting with 2b1d759e9ef87a385d86f4496f81777550173d4ed9111469e761941046d04c7f not found: ID does not exist"
Sep 18 20:34:18.934310 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.934293    2577 scope.go:117] "RemoveContainer" containerID="42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a"
Sep 18 20:34:18.934558 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:34:18.934539    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a\": container with ID starting with 42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a not found: ID does not exist" containerID="42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a"
Sep 18 20:34:18.934633 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.934563    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a"} err="failed to get container status \"42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a\": rpc error: code = NotFound desc = could not find container \"42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a\": container with ID starting with 42ecb59ee671bb67c5a2fe2659f54fab78fe0ea8b3beed0e292736c63f70fa2a not found: ID does not exist"
Sep 18 20:34:18.935901 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.935885    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af964572-fc1b-4917-aff8-5e95f8736d67-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:18.935948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.935905    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:18.935948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.935917    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af964572-fc1b-4917-aff8-5e95f8736d67-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:18.935948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.935925    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6vd\" (UniqueName: \"kubernetes.io/projected/af964572-fc1b-4917-aff8-5e95f8736d67-kube-api-access-ql6vd\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:18.935948 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:18.935934    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af964572-fc1b-4917-aff8-5e95f8736d67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:19.243145 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:19.243068    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" path="/var/lib/kubelet/pods/af964572-fc1b-4917-aff8-5e95f8736d67/volumes"
Sep 18 20:34:21.602557 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.602532    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:34:21.660327 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.660296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zhpj\" (UniqueName: \"kubernetes.io/projected/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-api-access-7zhpj\") pod \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") "
Sep 18 20:34:21.660502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.660352    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kserve-provision-location\") pod \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") "
Sep 18 20:34:21.660502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.660377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-cabundle-cert\") pod \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") "
Sep 18 20:34:21.660502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.660409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-rbac-proxy-sar-config\") pod \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") "
Sep 18 20:34:21.660785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.660690    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" (UID: "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:34:21.660904 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.660746    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" (UID: "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:34:21.660904 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.660830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" (UID: "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:34:21.662411 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.662387    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-api-access-7zhpj" (OuterVolumeSpecName: "kube-api-access-7zhpj") pod "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" (UID: "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0"). InnerVolumeSpecName "kube-api-access-7zhpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:34:21.761316 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.761283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-proxy-tls\") pod \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\" (UID: \"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0\") "
Sep 18 20:34:21.761535 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.761515    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:21.761585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.761541    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:21.761585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.761558    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:21.761585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.761571    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhpj\" (UniqueName: \"kubernetes.io/projected/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-kube-api-access-7zhpj\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:21.763319 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.763294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" (UID: "bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:34:21.862247 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.862212    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:34:21.919440 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.919357    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerID="f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578" exitCode=0
Sep 18 20:34:21.919440 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.919409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" event={"ID":"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0","Type":"ContainerDied","Data":"f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578"}
Sep 18 20:34:21.919626 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.919457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv" event={"ID":"bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0","Type":"ContainerDied","Data":"6d3b8b94717ca5fa3925808d5d01edf44a0476c5c298ef61c9352f4d8234fee9"}
Sep 18 20:34:21.919626 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.919416    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"
Sep 18 20:34:21.919626 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.919503    2577 scope.go:117] "RemoveContainer" containerID="930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a"
Sep 18 20:34:21.928175 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.928155    2577 scope.go:117] "RemoveContainer" containerID="f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578"
Sep 18 20:34:21.935837 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.935815    2577 scope.go:117] "RemoveContainer" containerID="bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34"
Sep 18 20:34:21.941242 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.941218    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"]
Sep 18 20:34:21.943461 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.943440    2577 scope.go:117] "RemoveContainer" containerID="930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a"
Sep 18 20:34:21.943895 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:34:21.943871    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a\": container with ID starting with 930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a not found: ID does not exist" containerID="930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a"
Sep 18 20:34:21.943978 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.943902    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a"} err="failed to get container status \"930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a\": rpc error: code = NotFound desc = could not find container \"930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a\": container with ID starting with 930c4fa3fdc3829a15442ee70199a16700007b3ec10fc5fdfad1f4e01fe9f59a not found: ID does not exist"
Sep 18 20:34:21.943978 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.943921    2577 scope.go:117] "RemoveContainer" containerID="f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578"
Sep 18 20:34:21.944182 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:34:21.944166    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578\": container with ID starting with f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578 not found: ID does not exist" containerID="f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578"
Sep 18 20:34:21.944219 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.944189    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578"} err="failed to get container status \"f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578\": rpc error: code = NotFound desc = could not find container \"f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578\": container with ID starting with f45b7c90e586b71d8479697dd2a612dfadc6f4dfb6e200569b53ae68871fa578 not found: ID does not exist"
Sep 18 20:34:21.944219 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.944203    2577 scope.go:117] "RemoveContainer" containerID="bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34"
Sep 18 20:34:21.944462 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:34:21.944446    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34\": container with ID starting with bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34 not found: ID does not exist" containerID="bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34"
Sep 18 20:34:21.944563 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.944467    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34"} err="failed to get container status \"bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34\": rpc error: code = NotFound desc = could not find container \"bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34\": container with ID starting with bf06f498a3308e48caf4f2948c6d04f27a59cba4911f66d535cca79e49452a34 not found: ID does not exist"
Sep 18 20:34:21.944912 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:21.944894    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-5mdrv"]
Sep 18 20:34:23.242898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:23.242866    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" path="/var/lib/kubelet/pods/bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0/volumes"
Sep 18 20:34:24.602188 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:24.602154    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:34:25.606709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:25.606678    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:34:29.242840 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:29.242803    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 20:34:39.243537 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:39.243467    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:34:51.529779 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.529739    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"]
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530037    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530048    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530058    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530063    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530078    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530083    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530090    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="storage-initializer"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530096    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="storage-initializer"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530102    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530108    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530114    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="storage-initializer"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530119    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="storage-initializer"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530129    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530134    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530138    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530143    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530148    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530153    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530160    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="storage-initializer"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530166    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="storage-initializer"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530210    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kserve-container"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530218    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b53ac6a4-55b4-4347-b75c-fae1ec83e52b" containerName="model-chainer-raw-12fdc"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530226    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530232    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af964572-fc1b-4917-aff8-5e95f8736d67" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530238    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kube-rbac-proxy"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530246    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f17e442f-ac76-493b-a93f-18334ff6354a" containerName="kserve-container"
Sep 18 20:34:51.530617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.530252    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcbf9678-9ddf-4056-a6fc-0b8714a5b5c0" containerName="kserve-container"
Sep 18 20:34:51.533091 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.533074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:51.535486 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.535454    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-e4001-serving-cert\""
Sep 18 20:34:51.535586 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.535458    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-e4001-kube-rbac-proxy-sar-config\""
Sep 18 20:34:51.544497 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.544450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"]
Sep 18 20:34:51.582206 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.582181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6879a101-e67c-4b48-bfb6-0b4baef5baab-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e4001-788b888c8c-pdbzk\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:51.582310 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.582232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls\") pod \"model-chainer-raw-hpa-e4001-788b888c8c-pdbzk\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:51.683150 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.683109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls\") pod \"model-chainer-raw-hpa-e4001-788b888c8c-pdbzk\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:51.683294 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.683195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6879a101-e67c-4b48-bfb6-0b4baef5baab-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e4001-788b888c8c-pdbzk\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:51.683294 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:34:51.683255    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-serving-cert: secret "model-chainer-raw-hpa-e4001-serving-cert" not found
Sep 18 20:34:51.683415 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:34:51.683316    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls podName:6879a101-e67c-4b48-bfb6-0b4baef5baab nodeName:}" failed. No retries permitted until 2026-09-18 20:34:52.183300125 +0000 UTC m=+868.993086045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls") pod "model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" (UID: "6879a101-e67c-4b48-bfb6-0b4baef5baab") : secret "model-chainer-raw-hpa-e4001-serving-cert" not found
Sep 18 20:34:51.683826 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:51.683803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6879a101-e67c-4b48-bfb6-0b4baef5baab-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e4001-788b888c8c-pdbzk\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:52.187774 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:52.187732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls\") pod \"model-chainer-raw-hpa-e4001-788b888c8c-pdbzk\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:52.190119 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:52.190097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls\") pod \"model-chainer-raw-hpa-e4001-788b888c8c-pdbzk\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:52.442997 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:52.442909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:52.708762 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:52.708738    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"]
Sep 18 20:34:52.711262 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:34:52.711239    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6879a101_e67c_4b48_bfb6_0b4baef5baab.slice/crio-0bf2298fe14aef4208cfece75662cf86c4d2a2570b88aa2f024a634c0ff8f679 WatchSource:0}: Error finding container 0bf2298fe14aef4208cfece75662cf86c4d2a2570b88aa2f024a634c0ff8f679: Status 404 returned error can't find the container with id 0bf2298fe14aef4208cfece75662cf86c4d2a2570b88aa2f024a634c0ff8f679
Sep 18 20:34:53.037884 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:53.037785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" event={"ID":"6879a101-e67c-4b48-bfb6-0b4baef5baab","Type":"ContainerStarted","Data":"bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359"}
Sep 18 20:34:53.038030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:53.037973    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:34:53.038030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:53.038006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" event={"ID":"6879a101-e67c-4b48-bfb6-0b4baef5baab","Type":"ContainerStarted","Data":"0bf2298fe14aef4208cfece75662cf86c4d2a2570b88aa2f024a634c0ff8f679"}
Sep 18 20:34:53.054519 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:53.054458    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" podStartSLOduration=2.054443827 podStartE2EDuration="2.054443827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:34:53.052971959 +0000 UTC m=+869.862757900" watchObservedRunningTime="2026-09-18 20:34:53.054443827 +0000 UTC m=+869.864229769"
Sep 18 20:34:59.046977 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:34:59.046947    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:35:01.578347 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:01.578316    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"]
Sep 18 20:35:01.578859 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:01.578548    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001" containerID="cri-o://bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359" gracePeriod=30
Sep 18 20:35:01.666920 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:01.666888    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"]
Sep 18 20:35:01.667196 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:01.667164    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container" containerID="cri-o://3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7" gracePeriod=30
Sep 18 20:35:01.667196 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:01.667191    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kube-rbac-proxy" containerID="cri-o://9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34" gracePeriod=30
Sep 18 20:35:01.705031 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:01.705006    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"]
Sep 18 20:35:01.705315 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:01.705276    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container" containerID="cri-o://ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642" gracePeriod=30
Sep 18 20:35:01.705397 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:01.705334    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kube-rbac-proxy" containerID="cri-o://a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d" gracePeriod=30
Sep 18 20:35:02.073673 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:02.073645    2577 generic.go:356] "Generic (PLEG): container finished" podID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerID="a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d" exitCode=2
Sep 18 20:35:02.073839 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:02.073716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" event={"ID":"64527dec-25cb-4226-b2bb-f10738bb36ce","Type":"ContainerDied","Data":"a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d"}
Sep 18 20:35:02.075731 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:02.075709    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff123e27-496f-4b9e-8c81-834504e16726" containerID="9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34" exitCode=2
Sep 18 20:35:02.075842 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:02.075741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" event={"ID":"ff123e27-496f-4b9e-8c81-834504e16726","Type":"ContainerDied","Data":"9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34"}
Sep 18 20:35:04.045393 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:04.045356    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:04.597573 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:04.597534    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 18 20:35:05.539301 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.539278    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:35:05.600045 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.599974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-cabundle-cert\") pod \"64527dec-25cb-4226-b2bb-f10738bb36ce\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") "
Sep 18 20:35:05.600045 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.600020    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flxgl\" (UniqueName: \"kubernetes.io/projected/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-api-access-flxgl\") pod \"64527dec-25cb-4226-b2bb-f10738bb36ce\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") "
Sep 18 20:35:05.600251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.600059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-rbac-proxy-sar-config\") pod \"64527dec-25cb-4226-b2bb-f10738bb36ce\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") "
Sep 18 20:35:05.600251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.600078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64527dec-25cb-4226-b2bb-f10738bb36ce-kserve-provision-location\") pod \"64527dec-25cb-4226-b2bb-f10738bb36ce\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") "
Sep 18 20:35:05.600251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.600133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls\") pod \"64527dec-25cb-4226-b2bb-f10738bb36ce\" (UID: \"64527dec-25cb-4226-b2bb-f10738bb36ce\") "
Sep 18 20:35:05.600403 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.600364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64527dec-25cb-4226-b2bb-f10738bb36ce" (UID: "64527dec-25cb-4226-b2bb-f10738bb36ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:05.600455 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.600407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64527dec-25cb-4226-b2bb-f10738bb36ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64527dec-25cb-4226-b2bb-f10738bb36ce" (UID: "64527dec-25cb-4226-b2bb-f10738bb36ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:05.600455 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.600444    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64527dec-25cb-4226-b2bb-f10738bb36ce" (UID: "64527dec-25cb-4226-b2bb-f10738bb36ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:05.602208 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.602181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64527dec-25cb-4226-b2bb-f10738bb36ce" (UID: "64527dec-25cb-4226-b2bb-f10738bb36ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:35:05.602309 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.602208    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-api-access-flxgl" (OuterVolumeSpecName: "kube-api-access-flxgl") pod "64527dec-25cb-4226-b2bb-f10738bb36ce" (UID: "64527dec-25cb-4226-b2bb-f10738bb36ce"). InnerVolumeSpecName "kube-api-access-flxgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:35:05.700604 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.700580    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64527dec-25cb-4226-b2bb-f10738bb36ce-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:05.700604 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.700602    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:05.700758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.700611    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flxgl\" (UniqueName: \"kubernetes.io/projected/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-api-access-flxgl\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:05.700758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.700620    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64527dec-25cb-4226-b2bb-f10738bb36ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:05.700758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:05.700629    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64527dec-25cb-4226-b2bb-f10738bb36ce-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:06.093002 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.092967    2577 generic.go:356] "Generic (PLEG): container finished" podID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerID="ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642" exitCode=0
Sep 18 20:35:06.093161 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.093020    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"
Sep 18 20:35:06.093161 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.093046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" event={"ID":"64527dec-25cb-4226-b2bb-f10738bb36ce","Type":"ContainerDied","Data":"ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642"}
Sep 18 20:35:06.093161 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.093116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs" event={"ID":"64527dec-25cb-4226-b2bb-f10738bb36ce","Type":"ContainerDied","Data":"4d1c0bc74a3022d729e312f31d1b15fb9f26e23d4b2f010df27af38c97f6606e"}
Sep 18 20:35:06.093161 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.093153    2577 scope.go:117] "RemoveContainer" containerID="a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d"
Sep 18 20:35:06.102975 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.102015    2577 scope.go:117] "RemoveContainer" containerID="ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642"
Sep 18 20:35:06.113640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.113617    2577 scope.go:117] "RemoveContainer" containerID="cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442"
Sep 18 20:35:06.114925 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.114906    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"]
Sep 18 20:35:06.119816 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.119793    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e4001-predictor-6c4ddd485c-t4fgs"]
Sep 18 20:35:06.121857 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.121836    2577 scope.go:117] "RemoveContainer" containerID="a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d"
Sep 18 20:35:06.122106 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:35:06.122089    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d\": container with ID starting with a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d not found: ID does not exist" containerID="a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d"
Sep 18 20:35:06.122157 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.122113    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d"} err="failed to get container status \"a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d\": rpc error: code = NotFound desc = could not find container \"a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d\": container with ID starting with a553ee3aa91d90798d1178935ca977f6436fdeebc0d1f74edceb32e7b2bf773d not found: ID does not exist"
Sep 18 20:35:06.122157 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.122129    2577 scope.go:117] "RemoveContainer" containerID="ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642"
Sep 18 20:35:06.122367 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:35:06.122351    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642\": container with ID starting with ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642 not found: ID does not exist" containerID="ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642"
Sep 18 20:35:06.122413 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.122371    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642"} err="failed to get container status \"ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642\": rpc error: code = NotFound desc = could not find container \"ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642\": container with ID starting with ad113d99d2ac041f154a88afc2fb1fb31546bfa01d0f4e35e23a3dea18967642 not found: ID does not exist"
Sep 18 20:35:06.122413 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.122384    2577 scope.go:117] "RemoveContainer" containerID="cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442"
Sep 18 20:35:06.122626 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:35:06.122611    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442\": container with ID starting with cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442 not found: ID does not exist" containerID="cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442"
Sep 18 20:35:06.122677 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.122630    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442"} err="failed to get container status \"cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442\": rpc error: code = NotFound desc = could not find container \"cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442\": container with ID starting with cfab18ef70a059c8141b19023436d7b5a6c5c1128be0dcf8450016ea1694e442 not found: ID does not exist"
Sep 18 20:35:06.295297 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.295273    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:35:06.406140 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.406071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls\") pod \"ff123e27-496f-4b9e-8c81-834504e16726\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") "
Sep 18 20:35:06.406140 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.406123    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-kube-rbac-proxy-sar-config\") pod \"ff123e27-496f-4b9e-8c81-834504e16726\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") "
Sep 18 20:35:06.406301 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.406162    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-cabundle-cert\") pod \"ff123e27-496f-4b9e-8c81-834504e16726\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") "
Sep 18 20:35:06.406301 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.406227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff123e27-496f-4b9e-8c81-834504e16726-kserve-provision-location\") pod \"ff123e27-496f-4b9e-8c81-834504e16726\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") "
Sep 18 20:35:06.406301 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.406276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvjlf\" (UniqueName: \"kubernetes.io/projected/ff123e27-496f-4b9e-8c81-834504e16726-kube-api-access-jvjlf\") pod \"ff123e27-496f-4b9e-8c81-834504e16726\" (UID: \"ff123e27-496f-4b9e-8c81-834504e16726\") "
Sep 18 20:35:06.406543 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.406521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff123e27-496f-4b9e-8c81-834504e16726-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff123e27-496f-4b9e-8c81-834504e16726" (UID: "ff123e27-496f-4b9e-8c81-834504e16726"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:06.406543 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.406533    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff123e27-496f-4b9e-8c81-834504e16726" (UID: "ff123e27-496f-4b9e-8c81-834504e16726"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:06.406677 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.406549    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff123e27-496f-4b9e-8c81-834504e16726" (UID: "ff123e27-496f-4b9e-8c81-834504e16726"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:06.408164 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.408139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff123e27-496f-4b9e-8c81-834504e16726" (UID: "ff123e27-496f-4b9e-8c81-834504e16726"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:35:06.408293 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.408275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff123e27-496f-4b9e-8c81-834504e16726-kube-api-access-jvjlf" (OuterVolumeSpecName: "kube-api-access-jvjlf") pod "ff123e27-496f-4b9e-8c81-834504e16726" (UID: "ff123e27-496f-4b9e-8c81-834504e16726"). InnerVolumeSpecName "kube-api-access-jvjlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:35:06.507310 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.507284    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff123e27-496f-4b9e-8c81-834504e16726-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:06.507310 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.507308    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjlf\" (UniqueName: \"kubernetes.io/projected/ff123e27-496f-4b9e-8c81-834504e16726-kube-api-access-jvjlf\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:06.507460 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.507317    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff123e27-496f-4b9e-8c81-834504e16726-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:06.507460 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.507325    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:06.507460 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:06.507335    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff123e27-496f-4b9e-8c81-834504e16726-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:07.099205 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.099131    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff123e27-496f-4b9e-8c81-834504e16726" containerID="3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7" exitCode=0
Sep 18 20:35:07.099205 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.099175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" event={"ID":"ff123e27-496f-4b9e-8c81-834504e16726","Type":"ContainerDied","Data":"3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7"}
Sep 18 20:35:07.099205 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.099183    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"
Sep 18 20:35:07.099715 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.099247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5" event={"ID":"ff123e27-496f-4b9e-8c81-834504e16726","Type":"ContainerDied","Data":"7fa908d7c85ce6a67cd6699c35643262304cf3e7262dcb57dca3d6d6cee436ca"}
Sep 18 20:35:07.099715 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.099275    2577 scope.go:117] "RemoveContainer" containerID="9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34"
Sep 18 20:35:07.107698 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.107680    2577 scope.go:117] "RemoveContainer" containerID="3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7"
Sep 18 20:35:07.114730 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.114712    2577 scope.go:117] "RemoveContainer" containerID="0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56"
Sep 18 20:35:07.121502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.121466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"]
Sep 18 20:35:07.121845 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.121828    2577 scope.go:117] "RemoveContainer" containerID="9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34"
Sep 18 20:35:07.122067 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:35:07.122051    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34\": container with ID starting with 9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34 not found: ID does not exist" containerID="9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34"
Sep 18 20:35:07.122109 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.122073    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34"} err="failed to get container status \"9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34\": rpc error: code = NotFound desc = could not find container \"9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34\": container with ID starting with 9fd1307bac5a658e5ecbc680c080439e1f52d0cfc4b2a75c992082280e566d34 not found: ID does not exist"
Sep 18 20:35:07.122109 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.122088    2577 scope.go:117] "RemoveContainer" containerID="3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7"
Sep 18 20:35:07.122322 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:35:07.122307    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7\": container with ID starting with 3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7 not found: ID does not exist" containerID="3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7"
Sep 18 20:35:07.122387 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.122324    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7"} err="failed to get container status \"3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7\": rpc error: code = NotFound desc = could not find container \"3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7\": container with ID starting with 3653f2b3d5f733cca97d30bacffb920d99ad2128b698be54170228b1943c8cd7 not found: ID does not exist"
Sep 18 20:35:07.122387 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.122336    2577 scope.go:117] "RemoveContainer" containerID="0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56"
Sep 18 20:35:07.122570 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:35:07.122555    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56\": container with ID starting with 0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56 not found: ID does not exist" containerID="0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56"
Sep 18 20:35:07.122621 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.122572    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56"} err="failed to get container status \"0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56\": rpc error: code = NotFound desc = could not find container \"0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56\": container with ID starting with 0b00fa3baf394e06dafa9cdf1a1b2bb377d9c48486b29620399f4f9f178faf56 not found: ID does not exist"
Sep 18 20:35:07.125862 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.125842    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e4001-predictor-fb8d76f98-8z6p5"]
Sep 18 20:35:07.242402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.242374    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" path="/var/lib/kubelet/pods/64527dec-25cb-4226-b2bb-f10738bb36ce/volumes"
Sep 18 20:35:07.242879 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:07.242864    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff123e27-496f-4b9e-8c81-834504e16726" path="/var/lib/kubelet/pods/ff123e27-496f-4b9e-8c81-834504e16726/volumes"
Sep 18 20:35:09.046136 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:09.046101    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:11.820244 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820205    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"]
Sep 18 20:35:11.820712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820647    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container"
Sep 18 20:35:11.820712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820665    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container"
Sep 18 20:35:11.820712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820686    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="storage-initializer"
Sep 18 20:35:11.820712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820694    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="storage-initializer"
Sep 18 20:35:11.820712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820704    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="storage-initializer"
Sep 18 20:35:11.820712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820714    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="storage-initializer"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820733    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kube-rbac-proxy"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820742    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kube-rbac-proxy"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820754    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kube-rbac-proxy"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820762    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kube-rbac-proxy"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820780    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820788    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820875    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kube-rbac-proxy"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820890    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kserve-container"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820899    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff123e27-496f-4b9e-8c81-834504e16726" containerName="kserve-container"
Sep 18 20:35:11.821030 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.820908    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="64527dec-25cb-4226-b2bb-f10738bb36ce" containerName="kube-rbac-proxy"
Sep 18 20:35:11.824089 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.824069    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:11.826462 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.826442    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-ad25b-kube-rbac-proxy-sar-config\""
Sep 18 20:35:11.826684 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.826662    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-ad25b-predictor-serving-cert\""
Sep 18 20:35:11.826684 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.826678    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:35:11.826875 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.826682    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:35:11.826875 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.826784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:35:11.835600 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.835581    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"]
Sep 18 20:35:11.950766 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.950732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-cabundle-cert\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:11.950932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.950773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kserve-provision-location\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:11.950932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.950802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-proxy-tls\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:11.950932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.950830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:11.950932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:11.950886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwnss\" (UniqueName: \"kubernetes.io/projected/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-api-access-vwnss\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.052058 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.052019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-proxy-tls\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.052058 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.052062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.052293 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.052108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwnss\" (UniqueName: \"kubernetes.io/projected/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-api-access-vwnss\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.052293 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.052145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-cabundle-cert\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.052293 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.052167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kserve-provision-location\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.052758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.052732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kserve-provision-location\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.052887 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.052873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-cabundle-cert\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.052941 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.052875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.054638 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.054616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-proxy-tls\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.060981 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.060960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwnss\" (UniqueName: \"kubernetes.io/projected/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-api-access-vwnss\") pod \"isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.134566 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.134541    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:12.409311 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:12.409239    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"]
Sep 18 20:35:12.412313 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:35:12.412285    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ef0f22_51c1_4b5e_85c0_7e89a2c3e303.slice/crio-1919dae09d933d67634e219d9137a04ecc28e68212deeb6f530a91c0cac9462f WatchSource:0}: Error finding container 1919dae09d933d67634e219d9137a04ecc28e68212deeb6f530a91c0cac9462f: Status 404 returned error can't find the container with id 1919dae09d933d67634e219d9137a04ecc28e68212deeb6f530a91c0cac9462f
Sep 18 20:35:13.125459 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:13.125427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerStarted","Data":"55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce"}
Sep 18 20:35:13.125824 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:13.125517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerStarted","Data":"1919dae09d933d67634e219d9137a04ecc28e68212deeb6f530a91c0cac9462f"}
Sep 18 20:35:14.046081 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:14.046039    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:14.046246 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:14.046134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:35:17.143715 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:17.143681    2577 generic.go:356] "Generic (PLEG): container finished" podID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerID="55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce" exitCode=0
Sep 18 20:35:17.144077 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:17.143758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerDied","Data":"55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce"}
Sep 18 20:35:18.149750 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.149715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerStarted","Data":"fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c"}
Sep 18 20:35:18.150159 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.149780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerStarted","Data":"26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a"}
Sep 18 20:35:18.150159 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.149805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerStarted","Data":"870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b"}
Sep 18 20:35:18.150159 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.150093    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:18.150159 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.150142    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:18.150319 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.150166    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:18.151675 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.151644    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:35:18.152343 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.152322    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:18.172296 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:18.172248    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podStartSLOduration=3.269173586 podStartE2EDuration="7.172236249s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.903062663s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:35:18.170824856 +0000 UTC m=+894.980610834" watchObservedRunningTime="2026-09-18 20:35:18.172236249 +0000 UTC m=+894.982022190"
Sep 18 20:35:19.046125 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:19.046092    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:19.154245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:19.154213    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:35:19.154651 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:19.154631    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:24.045242 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:24.045203    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:24.158589 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:24.158557    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:35:24.159076 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:24.159047    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:35:24.159383 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:24.159360    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:29.046236 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:29.046190    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:31.761847 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:31.761824    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:35:31.903613 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:31.903535    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6879a101-e67c-4b48-bfb6-0b4baef5baab-openshift-service-ca-bundle\") pod \"6879a101-e67c-4b48-bfb6-0b4baef5baab\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") "
Sep 18 20:35:31.903765 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:31.903632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls\") pod \"6879a101-e67c-4b48-bfb6-0b4baef5baab\" (UID: \"6879a101-e67c-4b48-bfb6-0b4baef5baab\") "
Sep 18 20:35:31.903962 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:31.903933    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6879a101-e67c-4b48-bfb6-0b4baef5baab-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6879a101-e67c-4b48-bfb6-0b4baef5baab" (UID: "6879a101-e67c-4b48-bfb6-0b4baef5baab"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:31.905639 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:31.905619    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6879a101-e67c-4b48-bfb6-0b4baef5baab" (UID: "6879a101-e67c-4b48-bfb6-0b4baef5baab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:35:32.004120 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.004086    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6879a101-e67c-4b48-bfb6-0b4baef5baab-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:32.004120 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.004118    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6879a101-e67c-4b48-bfb6-0b4baef5baab-openshift-service-ca-bundle\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:35:32.203432 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.203341    2577 generic.go:356] "Generic (PLEG): container finished" podID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerID="bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359" exitCode=137
Sep 18 20:35:32.203432 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.203382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"
Sep 18 20:35:32.203432 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.203402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" event={"ID":"6879a101-e67c-4b48-bfb6-0b4baef5baab","Type":"ContainerDied","Data":"bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359"}
Sep 18 20:35:32.203738 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.203469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk" event={"ID":"6879a101-e67c-4b48-bfb6-0b4baef5baab","Type":"ContainerDied","Data":"0bf2298fe14aef4208cfece75662cf86c4d2a2570b88aa2f024a634c0ff8f679"}
Sep 18 20:35:32.203738 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.203518    2577 scope.go:117] "RemoveContainer" containerID="bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359"
Sep 18 20:35:32.211531 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.211515    2577 scope.go:117] "RemoveContainer" containerID="bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359"
Sep 18 20:35:32.211802 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:35:32.211779    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359\": container with ID starting with bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359 not found: ID does not exist" containerID="bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359"
Sep 18 20:35:32.211903 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.211809    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359"} err="failed to get container status \"bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359\": rpc error: code = NotFound desc = could not find container \"bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359\": container with ID starting with bde046bbe89984dcdc89888e281b2867442b7fd1d88aebfdce7a09197f8d9359 not found: ID does not exist"
Sep 18 20:35:32.225108 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.225083    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"]
Sep 18 20:35:32.229267 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:32.229243    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e4001-788b888c8c-pdbzk"]
Sep 18 20:35:33.242602 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:33.242570    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" path="/var/lib/kubelet/pods/6879a101-e67c-4b48-bfb6-0b4baef5baab/volumes"
Sep 18 20:35:34.159070 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:34.159032    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:35:34.159538 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:34.159516    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:44.159909 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:44.159869    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:35:44.160362 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:44.160342    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:54.159142 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:54.159097    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:35:54.159695 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:35:54.159611    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:36:04.159111 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:04.159061    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:36:04.161317 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:04.159533    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:36:11.407294 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.407258    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"]
Sep 18 20:36:11.407681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.407575    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001"
Sep 18 20:36:11.407681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.407586    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001"
Sep 18 20:36:11.407681 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.407647    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6879a101-e67c-4b48-bfb6-0b4baef5baab" containerName="model-chainer-raw-hpa-e4001"
Sep 18 20:36:11.410697 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.410680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.413025 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.413003    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-878aa9-kube-rbac-proxy-sar-config\""
Sep 18 20:36:11.413139 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.413052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-878aa9\""
Sep 18 20:36:11.413139 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.413051    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-878aa9-predictor-serving-cert\""
Sep 18 20:36:11.413432 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.413414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-878aa9-dockercfg-br2zm\""
Sep 18 20:36:11.421535 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.421515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"]
Sep 18 20:36:11.501590 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.501551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgm8\" (UniqueName: \"kubernetes.io/projected/60a59189-536a-404a-8e54-46efddf1a672-kube-api-access-8pgm8\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.501590 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.501597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-cabundle-cert\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.501792 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.501676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.501792 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.501720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.501792 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.501766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a59189-536a-404a-8e54-46efddf1a672-kserve-provision-location\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.602277 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.602235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.602506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.602294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a59189-536a-404a-8e54-46efddf1a672-kserve-provision-location\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.602506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.602339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgm8\" (UniqueName: \"kubernetes.io/projected/60a59189-536a-404a-8e54-46efddf1a672-kube-api-access-8pgm8\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.602506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.602373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-cabundle-cert\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.602506 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:36:11.602394    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-serving-cert: secret "isvc-secondary-878aa9-predictor-serving-cert" not found
Sep 18 20:36:11.602506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.602427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.602506 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:36:11.602463    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls podName:60a59189-536a-404a-8e54-46efddf1a672 nodeName:}" failed. No retries permitted until 2026-09-18 20:36:12.10244699 +0000 UTC m=+948.912232910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls") pod "isvc-secondary-878aa9-predictor-7879cff9c6-hcm96" (UID: "60a59189-536a-404a-8e54-46efddf1a672") : secret "isvc-secondary-878aa9-predictor-serving-cert" not found
Sep 18 20:36:11.602872 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.602747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a59189-536a-404a-8e54-46efddf1a672-kserve-provision-location\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.603103 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.603081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.603189 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.603167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-cabundle-cert\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:11.610847 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:11.610826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgm8\" (UniqueName: \"kubernetes.io/projected/60a59189-536a-404a-8e54-46efddf1a672-kube-api-access-8pgm8\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:12.107114 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:12.107073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:12.109506 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:12.109465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls\") pod \"isvc-secondary-878aa9-predictor-7879cff9c6-hcm96\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") " pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:12.321384 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:12.321332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:12.590316 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:12.590286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"]
Sep 18 20:36:12.593840 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:36:12.593811    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a59189_536a_404a_8e54_46efddf1a672.slice/crio-4af031b9257912880a5e17babcbbd2c76a0d9caeb7f56db44d654ef67ee4d481 WatchSource:0}: Error finding container 4af031b9257912880a5e17babcbbd2c76a0d9caeb7f56db44d654ef67ee4d481: Status 404 returned error can't find the container with id 4af031b9257912880a5e17babcbbd2c76a0d9caeb7f56db44d654ef67ee4d481
Sep 18 20:36:13.360063 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:13.360029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96" event={"ID":"60a59189-536a-404a-8e54-46efddf1a672","Type":"ContainerStarted","Data":"ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9"}
Sep 18 20:36:13.360237 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:13.360084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96" event={"ID":"60a59189-536a-404a-8e54-46efddf1a672","Type":"ContainerStarted","Data":"4af031b9257912880a5e17babcbbd2c76a0d9caeb7f56db44d654ef67ee4d481"}
Sep 18 20:36:14.159632 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:14.159594    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:36:14.160064 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:14.160041    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:36:18.382095 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:18.382068    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_60a59189-536a-404a-8e54-46efddf1a672/storage-initializer/0.log"
Sep 18 20:36:18.382449 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:18.382105    2577 generic.go:356] "Generic (PLEG): container finished" podID="60a59189-536a-404a-8e54-46efddf1a672" containerID="ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9" exitCode=1
Sep 18 20:36:18.382449 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:18.382168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96" event={"ID":"60a59189-536a-404a-8e54-46efddf1a672","Type":"ContainerDied","Data":"ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9"}
Sep 18 20:36:19.387374 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:19.387348    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_60a59189-536a-404a-8e54-46efddf1a672/storage-initializer/0.log"
Sep 18 20:36:19.387741 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:19.387391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96" event={"ID":"60a59189-536a-404a-8e54-46efddf1a672","Type":"ContainerStarted","Data":"6f2b731fbd60fdb4fbb4023bfa9488a90b37bf0ce6ed121f33eb0ccf470c6dcf"}
Sep 18 20:36:24.159267 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.159226    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:36:24.159765 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.159744    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:36:24.407815 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.407790    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_60a59189-536a-404a-8e54-46efddf1a672/storage-initializer/1.log"
Sep 18 20:36:24.408213 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.408196    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_60a59189-536a-404a-8e54-46efddf1a672/storage-initializer/0.log"
Sep 18 20:36:24.408290 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.408231    2577 generic.go:356] "Generic (PLEG): container finished" podID="60a59189-536a-404a-8e54-46efddf1a672" containerID="6f2b731fbd60fdb4fbb4023bfa9488a90b37bf0ce6ed121f33eb0ccf470c6dcf" exitCode=1
Sep 18 20:36:24.408335 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.408308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96" event={"ID":"60a59189-536a-404a-8e54-46efddf1a672","Type":"ContainerDied","Data":"6f2b731fbd60fdb4fbb4023bfa9488a90b37bf0ce6ed121f33eb0ccf470c6dcf"}
Sep 18 20:36:24.408381 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.408372    2577 scope.go:117] "RemoveContainer" containerID="ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9"
Sep 18 20:36:24.408721 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.408703    2577 scope.go:117] "RemoveContainer" containerID="ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9"
Sep 18 20:36:24.418425 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:36:24.418398    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_kserve-ci-e2e-test(60a59189-536a-404a-8e54-46efddf1a672)\"" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96" podUID="60a59189-536a-404a-8e54-46efddf1a672"
Sep 18 20:36:24.418761 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:36:24.418736    2577 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_kserve-ci-e2e-test_60a59189-536a-404a-8e54-46efddf1a672_0 in pod sandbox 4af031b9257912880a5e17babcbbd2c76a0d9caeb7f56db44d654ef67ee4d481 from index: no such id: 'ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9'" containerID="ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9"
Sep 18 20:36:24.418862 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:24.418768    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_kserve-ci-e2e-test_60a59189-536a-404a-8e54-46efddf1a672_0 in pod sandbox 4af031b9257912880a5e17babcbbd2c76a0d9caeb7f56db44d654ef67ee4d481 from index: no such id: 'ee5cee7df94818db140e20a94788c573a9bcb4915d16d5cfb90030c2825e3cb9'"
Sep 18 20:36:25.413562 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:25.413532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_60a59189-536a-404a-8e54-46efddf1a672/storage-initializer/1.log"
Sep 18 20:36:29.448885 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.448857    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"]
Sep 18 20:36:29.574413 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.574391    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_60a59189-536a-404a-8e54-46efddf1a672/storage-initializer/1.log"
Sep 18 20:36:29.577966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.577949    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:29.657269 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.657245    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pgm8\" (UniqueName: \"kubernetes.io/projected/60a59189-536a-404a-8e54-46efddf1a672-kube-api-access-8pgm8\") pod \"60a59189-536a-404a-8e54-46efddf1a672\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") "
Sep 18 20:36:29.657373 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.657291    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-kube-rbac-proxy-sar-config\") pod \"60a59189-536a-404a-8e54-46efddf1a672\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") "
Sep 18 20:36:29.657373 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.657346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls\") pod \"60a59189-536a-404a-8e54-46efddf1a672\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") "
Sep 18 20:36:29.657512 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.657395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a59189-536a-404a-8e54-46efddf1a672-kserve-provision-location\") pod \"60a59189-536a-404a-8e54-46efddf1a672\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") "
Sep 18 20:36:29.657512 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.657428    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-cabundle-cert\") pod \"60a59189-536a-404a-8e54-46efddf1a672\" (UID: \"60a59189-536a-404a-8e54-46efddf1a672\") "
Sep 18 20:36:29.657752 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.657719    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a59189-536a-404a-8e54-46efddf1a672-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60a59189-536a-404a-8e54-46efddf1a672" (UID: "60a59189-536a-404a-8e54-46efddf1a672"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:36:29.657836 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.657697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60a59189-536a-404a-8e54-46efddf1a672" (UID: "60a59189-536a-404a-8e54-46efddf1a672"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:36:29.657836 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.657816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "60a59189-536a-404a-8e54-46efddf1a672" (UID: "60a59189-536a-404a-8e54-46efddf1a672"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:36:29.659320 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.659296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a59189-536a-404a-8e54-46efddf1a672-kube-api-access-8pgm8" (OuterVolumeSpecName: "kube-api-access-8pgm8") pod "60a59189-536a-404a-8e54-46efddf1a672" (UID: "60a59189-536a-404a-8e54-46efddf1a672"). InnerVolumeSpecName "kube-api-access-8pgm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:36:29.659413 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.659384    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60a59189-536a-404a-8e54-46efddf1a672" (UID: "60a59189-536a-404a-8e54-46efddf1a672"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:36:29.758937 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.758868    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgm8\" (UniqueName: \"kubernetes.io/projected/60a59189-536a-404a-8e54-46efddf1a672-kube-api-access-8pgm8\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:29.758937 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.758901    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:29.758937 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.758911    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a59189-536a-404a-8e54-46efddf1a672-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:29.758937 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.758921    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a59189-536a-404a-8e54-46efddf1a672-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:29.758937 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:29.758930    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a59189-536a-404a-8e54-46efddf1a672-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:30.434299 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:30.434272    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-878aa9-predictor-7879cff9c6-hcm96_60a59189-536a-404a-8e54-46efddf1a672/storage-initializer/1.log"
Sep 18 20:36:30.434449 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:30.434369    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"
Sep 18 20:36:30.434545 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:30.434371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96" event={"ID":"60a59189-536a-404a-8e54-46efddf1a672","Type":"ContainerDied","Data":"4af031b9257912880a5e17babcbbd2c76a0d9caeb7f56db44d654ef67ee4d481"}
Sep 18 20:36:30.434545 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:30.434535    2577 scope.go:117] "RemoveContainer" containerID="6f2b731fbd60fdb4fbb4023bfa9488a90b37bf0ce6ed121f33eb0ccf470c6dcf"
Sep 18 20:36:30.472174 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:30.472152    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"]
Sep 18 20:36:30.476394 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:30.476373    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-878aa9-predictor-7879cff9c6-hcm96"]
Sep 18 20:36:31.244084 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:31.244053    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a59189-536a-404a-8e54-46efddf1a672" path="/var/lib/kubelet/pods/60a59189-536a-404a-8e54-46efddf1a672/volumes"
Sep 18 20:36:34.160300 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.160273    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:36:34.160794 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.160728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:36:34.711290 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.711260    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"]
Sep 18 20:36:34.711593 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.711580    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a59189-536a-404a-8e54-46efddf1a672" containerName="storage-initializer"
Sep 18 20:36:34.711662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.711594    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a59189-536a-404a-8e54-46efddf1a672" containerName="storage-initializer"
Sep 18 20:36:34.711662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.711646    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a59189-536a-404a-8e54-46efddf1a672" containerName="storage-initializer"
Sep 18 20:36:34.711662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.711654    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a59189-536a-404a-8e54-46efddf1a672" containerName="storage-initializer"
Sep 18 20:36:34.711816 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.711705    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a59189-536a-404a-8e54-46efddf1a672" containerName="storage-initializer"
Sep 18 20:36:34.711816 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.711710    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a59189-536a-404a-8e54-46efddf1a672" containerName="storage-initializer"
Sep 18 20:36:34.715732 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.715713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.718440 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.718420    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-fbd03a\""
Sep 18 20:36:34.718550 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.718420    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fbd03a-kube-rbac-proxy-sar-config\""
Sep 18 20:36:34.718550 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.718539    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-fbd03a-dockercfg-wgwlt\""
Sep 18 20:36:34.718756 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.718739    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fbd03a-predictor-serving-cert\""
Sep 18 20:36:34.727858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.727831    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"]
Sep 18 20:36:34.798710 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.798682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe10421-129d-411f-a478-5a4faa479972-proxy-tls\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.798839 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.798721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe10421-129d-411f-a478-5a4faa479972-kserve-provision-location\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.798839 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.798775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.798839 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.798820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-cabundle-cert\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.798971 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.798842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btn5\" (UniqueName: \"kubernetes.io/projected/1fe10421-129d-411f-a478-5a4faa479972-kube-api-access-2btn5\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.899262 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.899238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe10421-129d-411f-a478-5a4faa479972-proxy-tls\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.899418 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.899269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe10421-129d-411f-a478-5a4faa479972-kserve-provision-location\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.899418 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.899288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.899418 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.899311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-cabundle-cert\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.899418 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.899338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btn5\" (UniqueName: \"kubernetes.io/projected/1fe10421-129d-411f-a478-5a4faa479972-kube-api-access-2btn5\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.899727 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.899708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe10421-129d-411f-a478-5a4faa479972-kserve-provision-location\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.900022 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.900002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-cabundle-cert\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.900097 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.900071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.901763 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.901734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe10421-129d-411f-a478-5a4faa479972-proxy-tls\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:34.907011 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:34.906989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btn5\" (UniqueName: \"kubernetes.io/projected/1fe10421-129d-411f-a478-5a4faa479972-kube-api-access-2btn5\") pod \"isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:35.026524 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:35.026457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:35.318507 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:35.318467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"]
Sep 18 20:36:35.319991 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:36:35.319966    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe10421_129d_411f_a478_5a4faa479972.slice/crio-1910d38c262b99a72136f152573f7842c966025e28c2de70fb326a7589c78c2f WatchSource:0}: Error finding container 1910d38c262b99a72136f152573f7842c966025e28c2de70fb326a7589c78c2f: Status 404 returned error can't find the container with id 1910d38c262b99a72136f152573f7842c966025e28c2de70fb326a7589c78c2f
Sep 18 20:36:35.456048 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:35.456012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd" event={"ID":"1fe10421-129d-411f-a478-5a4faa479972","Type":"ContainerStarted","Data":"caadb26309632e4434a6cf3d1fbcd242ad85b23b61bcd73f211366e44a121079"}
Sep 18 20:36:35.456209 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:35.456069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd" event={"ID":"1fe10421-129d-411f-a478-5a4faa479972","Type":"ContainerStarted","Data":"1910d38c262b99a72136f152573f7842c966025e28c2de70fb326a7589c78c2f"}
Sep 18 20:36:40.476976 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:40.476950    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd_1fe10421-129d-411f-a478-5a4faa479972/storage-initializer/0.log"
Sep 18 20:36:40.477322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:40.476986    2577 generic.go:356] "Generic (PLEG): container finished" podID="1fe10421-129d-411f-a478-5a4faa479972" containerID="caadb26309632e4434a6cf3d1fbcd242ad85b23b61bcd73f211366e44a121079" exitCode=1
Sep 18 20:36:40.477322 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:40.477010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd" event={"ID":"1fe10421-129d-411f-a478-5a4faa479972","Type":"ContainerDied","Data":"caadb26309632e4434a6cf3d1fbcd242ad85b23b61bcd73f211366e44a121079"}
Sep 18 20:36:41.483189 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:41.483157    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd_1fe10421-129d-411f-a478-5a4faa479972/storage-initializer/0.log"
Sep 18 20:36:41.483580 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:41.483237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd" event={"ID":"1fe10421-129d-411f-a478-5a4faa479972","Type":"ContainerStarted","Data":"a7c4ebb980d0e6ca30da438d736eba37fbe6dd8520618e00dc1c6fcf9321547a"}
Sep 18 20:36:44.689954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:44.689919    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"]
Sep 18 20:36:44.690351 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:44.690105    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd" podUID="1fe10421-129d-411f-a478-5a4faa479972" containerName="storage-initializer" containerID="cri-o://a7c4ebb980d0e6ca30da438d736eba37fbe6dd8520618e00dc1c6fcf9321547a" gracePeriod=30
Sep 18 20:36:45.499056 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.499027    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd_1fe10421-129d-411f-a478-5a4faa479972/storage-initializer/1.log"
Sep 18 20:36:45.499450 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.499427    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd_1fe10421-129d-411f-a478-5a4faa479972/storage-initializer/0.log"
Sep 18 20:36:45.499598 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.499466    2577 generic.go:356] "Generic (PLEG): container finished" podID="1fe10421-129d-411f-a478-5a4faa479972" containerID="a7c4ebb980d0e6ca30da438d736eba37fbe6dd8520618e00dc1c6fcf9321547a" exitCode=1
Sep 18 20:36:45.499598 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.499502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd" event={"ID":"1fe10421-129d-411f-a478-5a4faa479972","Type":"ContainerDied","Data":"a7c4ebb980d0e6ca30da438d736eba37fbe6dd8520618e00dc1c6fcf9321547a"}
Sep 18 20:36:45.499598 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.499584    2577 scope.go:117] "RemoveContainer" containerID="caadb26309632e4434a6cf3d1fbcd242ad85b23b61bcd73f211366e44a121079"
Sep 18 20:36:45.530932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.530916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd_1fe10421-129d-411f-a478-5a4faa479972/storage-initializer/1.log"
Sep 18 20:36:45.534235 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.534220    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:45.687230 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe10421-129d-411f-a478-5a4faa479972-proxy-tls\") pod \"1fe10421-129d-411f-a478-5a4faa479972\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") "
Sep 18 20:36:45.687388 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687239    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-cabundle-cert\") pod \"1fe10421-129d-411f-a478-5a4faa479972\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") "
Sep 18 20:36:45.687388 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe10421-129d-411f-a478-5a4faa479972-kserve-provision-location\") pod \"1fe10421-129d-411f-a478-5a4faa479972\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") "
Sep 18 20:36:45.687388 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687367    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-kube-rbac-proxy-sar-config\") pod \"1fe10421-129d-411f-a478-5a4faa479972\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") "
Sep 18 20:36:45.687583 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687416    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2btn5\" (UniqueName: \"kubernetes.io/projected/1fe10421-129d-411f-a478-5a4faa479972-kube-api-access-2btn5\") pod \"1fe10421-129d-411f-a478-5a4faa479972\" (UID: \"1fe10421-129d-411f-a478-5a4faa479972\") "
Sep 18 20:36:45.687583 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687509    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe10421-129d-411f-a478-5a4faa479972-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1fe10421-129d-411f-a478-5a4faa479972" (UID: "1fe10421-129d-411f-a478-5a4faa479972"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:36:45.687698 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687673    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1fe10421-129d-411f-a478-5a4faa479972" (UID: "1fe10421-129d-411f-a478-5a4faa479972"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:36:45.687740 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687702    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe10421-129d-411f-a478-5a4faa479972-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:45.687772 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.687728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1fe10421-129d-411f-a478-5a4faa479972" (UID: "1fe10421-129d-411f-a478-5a4faa479972"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:36:45.689247 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.689229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe10421-129d-411f-a478-5a4faa479972-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1fe10421-129d-411f-a478-5a4faa479972" (UID: "1fe10421-129d-411f-a478-5a4faa479972"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:36:45.689335 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.689318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe10421-129d-411f-a478-5a4faa479972-kube-api-access-2btn5" (OuterVolumeSpecName: "kube-api-access-2btn5") pod "1fe10421-129d-411f-a478-5a4faa479972" (UID: "1fe10421-129d-411f-a478-5a4faa479972"). InnerVolumeSpecName "kube-api-access-2btn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:36:45.789038 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.789012    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe10421-129d-411f-a478-5a4faa479972-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:45.789038 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.789035    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:45.789038 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.789045    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe10421-129d-411f-a478-5a4faa479972-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:45.789425 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:45.789055    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btn5\" (UniqueName: \"kubernetes.io/projected/1fe10421-129d-411f-a478-5a4faa479972-kube-api-access-2btn5\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:36:46.504991 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:46.504965    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd_1fe10421-129d-411f-a478-5a4faa479972/storage-initializer/1.log"
Sep 18 20:36:46.505149 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:46.505068    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"
Sep 18 20:36:46.505192 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:46.505062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd" event={"ID":"1fe10421-129d-411f-a478-5a4faa479972","Type":"ContainerDied","Data":"1910d38c262b99a72136f152573f7842c966025e28c2de70fb326a7589c78c2f"}
Sep 18 20:36:46.505228 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:46.505202    2577 scope.go:117] "RemoveContainer" containerID="a7c4ebb980d0e6ca30da438d736eba37fbe6dd8520618e00dc1c6fcf9321547a"
Sep 18 20:36:46.562803 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:46.562774    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"]
Sep 18 20:36:46.575093 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:46.575071    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbd03a-predictor-5d54576f86-sc8dd"]
Sep 18 20:36:47.243676 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.243640    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe10421-129d-411f-a478-5a4faa479972" path="/var/lib/kubelet/pods/1fe10421-129d-411f-a478-5a4faa479972/volumes"
Sep 18 20:36:47.466930 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.466898    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"]
Sep 18 20:36:47.467229 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.467201    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" containerID="cri-o://870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b" gracePeriod=30
Sep 18 20:36:47.467499 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.467306    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy" containerID="cri-o://26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a" gracePeriod=30
Sep 18 20:36:47.467650 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.467293    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" containerID="cri-o://fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c" gracePeriod=30
Sep 18 20:36:47.599330 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.599302    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"]
Sep 18 20:36:47.599685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.599668    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe10421-129d-411f-a478-5a4faa479972" containerName="storage-initializer"
Sep 18 20:36:47.599770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.599687    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe10421-129d-411f-a478-5a4faa479972" containerName="storage-initializer"
Sep 18 20:36:47.599770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.599702    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe10421-129d-411f-a478-5a4faa479972" containerName="storage-initializer"
Sep 18 20:36:47.599770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.599734    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe10421-129d-411f-a478-5a4faa479972" containerName="storage-initializer"
Sep 18 20:36:47.599929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.599829    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe10421-129d-411f-a478-5a4faa479972" containerName="storage-initializer"
Sep 18 20:36:47.599929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.599843    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe10421-129d-411f-a478-5a4faa479972" containerName="storage-initializer"
Sep 18 20:36:47.602884 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.602866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.606053 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.606021    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-2e756-kube-rbac-proxy-sar-config\""
Sep 18 20:36:47.606162 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.606021    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-2e756-predictor-serving-cert\""
Sep 18 20:36:47.616084 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.616063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"]
Sep 18 20:36:47.701407 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.701379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9848cb9c-0659-4770-9400-35c121b2ed51-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.701558 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.701410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.701558 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.701442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvtm\" (UniqueName: \"kubernetes.io/projected/9848cb9c-0659-4770-9400-35c121b2ed51-kube-api-access-zjvtm\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.701558 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.701504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-cabundle-cert\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.701558 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.701529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9848cb9c-0659-4770-9400-35c121b2ed51-proxy-tls\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.802112 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.802079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-cabundle-cert\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.802112 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.802115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9848cb9c-0659-4770-9400-35c121b2ed51-proxy-tls\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.802302 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.802289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9848cb9c-0659-4770-9400-35c121b2ed51-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.802351 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.802313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.802351 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.802345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvtm\" (UniqueName: \"kubernetes.io/projected/9848cb9c-0659-4770-9400-35c121b2ed51-kube-api-access-zjvtm\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.802659 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.802629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9848cb9c-0659-4770-9400-35c121b2ed51-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.802821 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.802803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-cabundle-cert\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.802972 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.802953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.804539 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.804521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9848cb9c-0659-4770-9400-35c121b2ed51-proxy-tls\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.811195 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.811175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvtm\" (UniqueName: \"kubernetes.io/projected/9848cb9c-0659-4770-9400-35c121b2ed51-kube-api-access-zjvtm\") pod \"isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:47.912508 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:47.912413    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:48.181601 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:48.181570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"]
Sep 18 20:36:48.183858 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:36:48.183834    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9848cb9c_0659_4770_9400_35c121b2ed51.slice/crio-3d1aef2c762769b295be21a9c1fc2cb031cee2c848a0768199b6aa3e14a9c9cb WatchSource:0}: Error finding container 3d1aef2c762769b295be21a9c1fc2cb031cee2c848a0768199b6aa3e14a9c9cb: Status 404 returned error can't find the container with id 3d1aef2c762769b295be21a9c1fc2cb031cee2c848a0768199b6aa3e14a9c9cb
Sep 18 20:36:48.515403 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:48.515305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" event={"ID":"9848cb9c-0659-4770-9400-35c121b2ed51","Type":"ContainerStarted","Data":"e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3"}
Sep 18 20:36:48.515403 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:48.515377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" event={"ID":"9848cb9c-0659-4770-9400-35c121b2ed51","Type":"ContainerStarted","Data":"3d1aef2c762769b295be21a9c1fc2cb031cee2c848a0768199b6aa3e14a9c9cb"}
Sep 18 20:36:48.518198 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:48.518173    2577 generic.go:356] "Generic (PLEG): container finished" podID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerID="26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a" exitCode=2
Sep 18 20:36:48.518359 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:48.518206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerDied","Data":"26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a"}
Sep 18 20:36:49.155367 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:49.155323    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 20:36:52.535249 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:52.535218    2577 generic.go:356] "Generic (PLEG): container finished" podID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerID="870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b" exitCode=0
Sep 18 20:36:52.535614 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:52.535285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerDied","Data":"870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b"}
Sep 18 20:36:52.536775 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:52.536747    2577 generic.go:356] "Generic (PLEG): container finished" podID="9848cb9c-0659-4770-9400-35c121b2ed51" containerID="e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3" exitCode=0
Sep 18 20:36:52.536898 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:52.536793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" event={"ID":"9848cb9c-0659-4770-9400-35c121b2ed51","Type":"ContainerDied","Data":"e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3"}
Sep 18 20:36:53.542223 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:53.542192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" event={"ID":"9848cb9c-0659-4770-9400-35c121b2ed51","Type":"ContainerStarted","Data":"678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947"}
Sep 18 20:36:53.542590 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:53.542253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" event={"ID":"9848cb9c-0659-4770-9400-35c121b2ed51","Type":"ContainerStarted","Data":"3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44"}
Sep 18 20:36:53.542590 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:53.542561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:53.542715 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:53.542696    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:53.544154 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:53.544128    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:36:53.564452 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:53.564410    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podStartSLOduration=2.869309475 podStartE2EDuration="6.56439674s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695087265s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:36:53.562406688 +0000 UTC m=+990.372192652" watchObservedRunningTime="2026-09-18 20:36:53.56439674 +0000 UTC m=+990.374182681"
Sep 18 20:36:54.154781 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:54.154744    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 20:36:54.159398 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:54.159361    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:36:54.161078 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:54.161059    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:36:54.546623 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:54.546525    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:36:59.155316 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:59.155271    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 20:36:59.155714 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:59.155402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:36:59.551298 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:59.551223    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:36:59.551786 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:36:59.551761    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:37:04.154578 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:04.154538    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 20:37:04.160344 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:04.160313    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:37:04.161592 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:04.161566    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:37:09.154527 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:09.154465    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 20:37:09.551737 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:09.551651    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:37:14.154406 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:14.154367    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 20:37:14.160053 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:14.160019    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:37:14.160167 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:14.160119    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:37:14.161263 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:14.161240    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:37:14.161373 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:14.161350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:37:17.609306 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.609280    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:37:17.635432 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.635395    2577 generic.go:356] "Generic (PLEG): container finished" podID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerID="fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c" exitCode=0
Sep 18 20:37:17.635634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.635461    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"
Sep 18 20:37:17.635634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.635470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerDied","Data":"fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c"}
Sep 18 20:37:17.635634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.635563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb" event={"ID":"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303","Type":"ContainerDied","Data":"1919dae09d933d67634e219d9137a04ecc28e68212deeb6f530a91c0cac9462f"}
Sep 18 20:37:17.635634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.635608    2577 scope.go:117] "RemoveContainer" containerID="fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c"
Sep 18 20:37:17.644223 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.644195    2577 scope.go:117] "RemoveContainer" containerID="26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a"
Sep 18 20:37:17.652970 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.652950    2577 scope.go:117] "RemoveContainer" containerID="870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b"
Sep 18 20:37:17.660016 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.659988    2577 scope.go:117] "RemoveContainer" containerID="55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce"
Sep 18 20:37:17.667185 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.667168    2577 scope.go:117] "RemoveContainer" containerID="fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c"
Sep 18 20:37:17.667448 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:37:17.667426    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c\": container with ID starting with fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c not found: ID does not exist" containerID="fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c"
Sep 18 20:37:17.667542 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.667460    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c"} err="failed to get container status \"fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c\": rpc error: code = NotFound desc = could not find container \"fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c\": container with ID starting with fcb1098f610d2e9eea49777d42071908c2a48a5d70ecde54eb023194da7ce68c not found: ID does not exist"
Sep 18 20:37:17.667542 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.667500    2577 scope.go:117] "RemoveContainer" containerID="26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a"
Sep 18 20:37:17.667773 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:37:17.667755    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a\": container with ID starting with 26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a not found: ID does not exist" containerID="26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a"
Sep 18 20:37:17.667817 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.667779    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a"} err="failed to get container status \"26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a\": rpc error: code = NotFound desc = could not find container \"26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a\": container with ID starting with 26ad09700dc8b59b8ce44b566493c879edebd2a11f8c824ac783867a3416da9a not found: ID does not exist"
Sep 18 20:37:17.667817 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.667794    2577 scope.go:117] "RemoveContainer" containerID="870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b"
Sep 18 20:37:17.668020 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:37:17.667998    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b\": container with ID starting with 870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b not found: ID does not exist" containerID="870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b"
Sep 18 20:37:17.668085 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.668028    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b"} err="failed to get container status \"870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b\": rpc error: code = NotFound desc = could not find container \"870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b\": container with ID starting with 870d98f546e89dba6c002f3c227fbac1b8b8b1672417c89b084db16aa5fcf41b not found: ID does not exist"
Sep 18 20:37:17.668085 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.668048    2577 scope.go:117] "RemoveContainer" containerID="55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce"
Sep 18 20:37:17.668316 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:37:17.668299    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce\": container with ID starting with 55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce not found: ID does not exist" containerID="55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce"
Sep 18 20:37:17.668355 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.668320    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce"} err="failed to get container status \"55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce\": rpc error: code = NotFound desc = could not find container \"55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce\": container with ID starting with 55abb44801f08732404aebdc6fe7111d962218bb455f813cd75fe5b6e3b5b8ce not found: ID does not exist"
Sep 18 20:37:17.741306 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.741209    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwnss\" (UniqueName: \"kubernetes.io/projected/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-api-access-vwnss\") pod \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") "
Sep 18 20:37:17.741306 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.741252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-proxy-tls\") pod \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") "
Sep 18 20:37:17.741306 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.741278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-rbac-proxy-sar-config\") pod \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") "
Sep 18 20:37:17.741632 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.741341    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-cabundle-cert\") pod \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") "
Sep 18 20:37:17.741632 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.741554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kserve-provision-location\") pod \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\" (UID: \"97ef0f22-51c1-4b5e-85c0-7e89a2c3e303\") "
Sep 18 20:37:17.741832 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.741739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" (UID: "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:37:17.741941 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.741804    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" (UID: "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:37:17.741941 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.741853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" (UID: "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:17.743940 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.743915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" (UID: "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:37:17.743940 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.743925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-api-access-vwnss" (OuterVolumeSpecName: "kube-api-access-vwnss") pod "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" (UID: "97ef0f22-51c1-4b5e-85c0-7e89a2c3e303"). InnerVolumeSpecName "kube-api-access-vwnss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:37:17.842617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.842577    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:37:17.842617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.842608    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwnss\" (UniqueName: \"kubernetes.io/projected/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-api-access-vwnss\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:37:17.842617 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.842618    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:37:17.842866 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.842634    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:37:17.842866 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.842645    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:37:17.959278 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.959244    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"]
Sep 18 20:37:17.966592 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:17.966568    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ad25b-predictor-6c76fc79b9-x49wb"]
Sep 18 20:37:19.243138 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:19.243108    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" path="/var/lib/kubelet/pods/97ef0f22-51c1-4b5e-85c0-7e89a2c3e303/volumes"
Sep 18 20:37:19.551942 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:19.551857    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:37:29.552122 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:29.552082    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:37:39.551702 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:39.551653    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:37:49.551916 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:49.551880    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:37:59.551974 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:37:59.551924    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:38:00.238860 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:38:00.238813    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:38:10.239111 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:38:10.239067    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:38:20.239147 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:38:20.239105    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:38:30.238688 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:38:30.238643    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:38:40.238937 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:38:40.238898    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:38:50.239299 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:38:50.239255    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:39:00.239304 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:00.239265    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:39:10.239718 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:10.239678    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:39:13.238823 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:13.238783    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:39:23.239369 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:23.239323    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:39:33.242638 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:33.242603    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:39:38.739446 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:38.739361    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"]
Sep 18 20:39:38.739928 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:38.739632    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" containerID="cri-o://3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44" gracePeriod=30
Sep 18 20:39:38.739928 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:38.739666    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kube-rbac-proxy" containerID="cri-o://678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947" gracePeriod=30
Sep 18 20:39:39.166196 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:39.166166    2577 generic.go:356] "Generic (PLEG): container finished" podID="9848cb9c-0659-4770-9400-35c121b2ed51" containerID="678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947" exitCode=2
Sep 18 20:39:39.166368 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:39.166235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" event={"ID":"9848cb9c-0659-4770-9400-35c121b2ed51","Type":"ContainerDied","Data":"678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947"}
Sep 18 20:39:39.546979 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:39.546889    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 18 20:39:43.238947 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.238909    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:39:43.934557 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934519    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"]
Sep 18 20:39:43.934929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934913    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy"
Sep 18 20:39:43.934929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934930    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934951    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="storage-initializer"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934957    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="storage-initializer"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934964    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934969    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934979    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.934984    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.935028    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kube-rbac-proxy"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.935037    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="agent"
Sep 18 20:39:43.935057 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.935044    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ef0f22-51c1-4b5e-85c0-7e89a2c3e303" containerName="kserve-container"
Sep 18 20:39:43.938082 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.938064    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:43.941835 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.941813    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 18 20:39:43.941918 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.941856    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 20:39:43.946557 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:43.946539    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"]
Sep 18 20:39:44.060365 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.060326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqsk\" (UniqueName: \"kubernetes.io/projected/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-api-access-8vqsk\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.060532 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.060384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.060532 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.060417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.060532 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.060464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.060661 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.060544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.069334 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.069310    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"]
Sep 18 20:39:44.072571 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.072557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.077059 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.077043    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 20:39:44.095522 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.095500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"]
Sep 18 20:39:44.161261 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.161421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.161421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.161421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.161625 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.161625 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqsk\" (UniqueName: \"kubernetes.io/projected/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-api-access-8vqsk\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.161625 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.161625 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxtwg\" (UniqueName: \"kubernetes.io/projected/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-api-access-hxtwg\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.161836 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.161836 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.161836 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.161684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.162265 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.162244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.162331 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.162271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.163785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.163769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.177418 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.177395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqsk\" (UniqueName: \"kubernetes.io/projected/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-api-access-8vqsk\") pod \"isvc-canary-create-v2-predictor-667f95d746-xfwx7\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.249305 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.249238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:44.262403 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.262377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.262540 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.262424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.262540 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.262463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.262672 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.262538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxtwg\" (UniqueName: \"kubernetes.io/projected/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-api-access-hxtwg\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.262672 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.262565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.262672 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:39:44.262573    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:39:44.262672 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:39:44.262646    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls podName:2ed73cd2-73de-4f8e-8ee7-df2fd81c2131 nodeName:}" failed. No retries permitted until 2026-09-18 20:39:44.762626041 +0000 UTC m=+1161.572411978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls") pod "isvc-canary-create-predictor-76b74f4d6c-z62mp" (UID: "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:39:44.263070 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.263048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.263264 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.263239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.263371 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.263353    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.271390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.271370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxtwg\" (UniqueName: \"kubernetes.io/projected/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-api-access-hxtwg\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.526178 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.526098    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"]
Sep 18 20:39:44.530526 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:39:44.530502    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a06112_ffac_4124_ba1e_ba9ebc52b4d0.slice/crio-58f70b73d405aec54ec62860f374b7acbabc2347687db8b8fa58af17cfbcedd5 WatchSource:0}: Error finding container 58f70b73d405aec54ec62860f374b7acbabc2347687db8b8fa58af17cfbcedd5: Status 404 returned error can't find the container with id 58f70b73d405aec54ec62860f374b7acbabc2347687db8b8fa58af17cfbcedd5
Sep 18 20:39:44.547360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.547332    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 18 20:39:44.766809 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.766771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.769010 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.768988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-z62mp\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:44.982848 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:44.982810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:45.189707 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:45.189677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" event={"ID":"41a06112-ffac-4124-ba1e-ba9ebc52b4d0","Type":"ContainerStarted","Data":"5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970"}
Sep 18 20:39:45.189892 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:45.189734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" event={"ID":"41a06112-ffac-4124-ba1e-ba9ebc52b4d0","Type":"ContainerStarted","Data":"58f70b73d405aec54ec62860f374b7acbabc2347687db8b8fa58af17cfbcedd5"}
Sep 18 20:39:45.259021 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:45.258997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"]
Sep 18 20:39:45.261042 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:39:45.261016    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed73cd2_73de_4f8e_8ee7_df2fd81c2131.slice/crio-5125ca833ac1f4f378bb3bac399d43e07b42bfc0a09069e8667363cdfacf96d6 WatchSource:0}: Error finding container 5125ca833ac1f4f378bb3bac399d43e07b42bfc0a09069e8667363cdfacf96d6: Status 404 returned error can't find the container with id 5125ca833ac1f4f378bb3bac399d43e07b42bfc0a09069e8667363cdfacf96d6
Sep 18 20:39:46.195219 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:46.195183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" event={"ID":"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131","Type":"ContainerStarted","Data":"51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903"}
Sep 18 20:39:46.195392 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:46.195243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" event={"ID":"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131","Type":"ContainerStarted","Data":"5125ca833ac1f4f378bb3bac399d43e07b42bfc0a09069e8667363cdfacf96d6"}
Sep 18 20:39:48.882834 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:48.882808    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:39:49.000825 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.000797    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9848cb9c-0659-4770-9400-35c121b2ed51-proxy-tls\") pod \"9848cb9c-0659-4770-9400-35c121b2ed51\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") "
Sep 18 20:39:49.000949 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.000882    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-cabundle-cert\") pod \"9848cb9c-0659-4770-9400-35c121b2ed51\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") "
Sep 18 20:39:49.000949 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.000908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjvtm\" (UniqueName: \"kubernetes.io/projected/9848cb9c-0659-4770-9400-35c121b2ed51-kube-api-access-zjvtm\") pod \"9848cb9c-0659-4770-9400-35c121b2ed51\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") "
Sep 18 20:39:49.000949 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.000930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9848cb9c-0659-4770-9400-35c121b2ed51-kserve-provision-location\") pod \"9848cb9c-0659-4770-9400-35c121b2ed51\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") "
Sep 18 20:39:49.000949 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.000946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-kube-rbac-proxy-sar-config\") pod \"9848cb9c-0659-4770-9400-35c121b2ed51\" (UID: \"9848cb9c-0659-4770-9400-35c121b2ed51\") "
Sep 18 20:39:49.001291 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.001257    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9848cb9c-0659-4770-9400-35c121b2ed51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9848cb9c-0659-4770-9400-35c121b2ed51" (UID: "9848cb9c-0659-4770-9400-35c121b2ed51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:39:49.001291 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.001282    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9848cb9c-0659-4770-9400-35c121b2ed51" (UID: "9848cb9c-0659-4770-9400-35c121b2ed51"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:39:49.001416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.001326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9848cb9c-0659-4770-9400-35c121b2ed51" (UID: "9848cb9c-0659-4770-9400-35c121b2ed51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:39:49.002729 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.002707    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9848cb9c-0659-4770-9400-35c121b2ed51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9848cb9c-0659-4770-9400-35c121b2ed51" (UID: "9848cb9c-0659-4770-9400-35c121b2ed51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:39:49.002918 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.002898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9848cb9c-0659-4770-9400-35c121b2ed51-kube-api-access-zjvtm" (OuterVolumeSpecName: "kube-api-access-zjvtm") pod "9848cb9c-0659-4770-9400-35c121b2ed51" (UID: "9848cb9c-0659-4770-9400-35c121b2ed51"). InnerVolumeSpecName "kube-api-access-zjvtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:39:49.102126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.102095    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:39:49.102126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.102121    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjvtm\" (UniqueName: \"kubernetes.io/projected/9848cb9c-0659-4770-9400-35c121b2ed51-kube-api-access-zjvtm\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:39:49.102126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.102131    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9848cb9c-0659-4770-9400-35c121b2ed51-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:39:49.102312 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.102140    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9848cb9c-0659-4770-9400-35c121b2ed51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:39:49.102312 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.102150    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9848cb9c-0659-4770-9400-35c121b2ed51-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:39:49.207737 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.207668    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerID="51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903" exitCode=0
Sep 18 20:39:49.207878 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.207743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" event={"ID":"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131","Type":"ContainerDied","Data":"51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903"}
Sep 18 20:39:49.209404 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.209383    2577 generic.go:356] "Generic (PLEG): container finished" podID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerID="5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970" exitCode=0
Sep 18 20:39:49.209505 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.209451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" event={"ID":"41a06112-ffac-4124-ba1e-ba9ebc52b4d0","Type":"ContainerDied","Data":"5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970"}
Sep 18 20:39:49.211613 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.211591    2577 generic.go:356] "Generic (PLEG): container finished" podID="9848cb9c-0659-4770-9400-35c121b2ed51" containerID="3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44" exitCode=0
Sep 18 20:39:49.211714 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.211620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" event={"ID":"9848cb9c-0659-4770-9400-35c121b2ed51","Type":"ContainerDied","Data":"3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44"}
Sep 18 20:39:49.211714 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.211635    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"
Sep 18 20:39:49.211714 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.211659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf" event={"ID":"9848cb9c-0659-4770-9400-35c121b2ed51","Type":"ContainerDied","Data":"3d1aef2c762769b295be21a9c1fc2cb031cee2c848a0768199b6aa3e14a9c9cb"}
Sep 18 20:39:49.211714 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.211696    2577 scope.go:117] "RemoveContainer" containerID="678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947"
Sep 18 20:39:49.221693 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.221678    2577 scope.go:117] "RemoveContainer" containerID="3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44"
Sep 18 20:39:49.230947 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.230931    2577 scope.go:117] "RemoveContainer" containerID="e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3"
Sep 18 20:39:49.239827 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.239800    2577 scope.go:117] "RemoveContainer" containerID="678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947"
Sep 18 20:39:49.240088 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:39:49.240069    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947\": container with ID starting with 678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947 not found: ID does not exist" containerID="678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947"
Sep 18 20:39:49.240174 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.240097    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947"} err="failed to get container status \"678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947\": rpc error: code = NotFound desc = could not find container \"678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947\": container with ID starting with 678a65b48dca79d48f3f300ae630115e5f76a00a59197ddc0b5322477ad84947 not found: ID does not exist"
Sep 18 20:39:49.240174 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.240119    2577 scope.go:117] "RemoveContainer" containerID="3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44"
Sep 18 20:39:49.240427 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:39:49.240378    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44\": container with ID starting with 3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44 not found: ID does not exist" containerID="3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44"
Sep 18 20:39:49.240427 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.240408    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44"} err="failed to get container status \"3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44\": rpc error: code = NotFound desc = could not find container \"3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44\": container with ID starting with 3a7e8daee06aacf5509f56da2c25f876aad3c6ad8bc0ec2a0e15276600327e44 not found: ID does not exist"
Sep 18 20:39:49.240573 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.240429    2577 scope.go:117] "RemoveContainer" containerID="e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3"
Sep 18 20:39:49.240877 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:39:49.240853    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3\": container with ID starting with e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3 not found: ID does not exist" containerID="e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3"
Sep 18 20:39:49.240932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.240885    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3"} err="failed to get container status \"e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3\": rpc error: code = NotFound desc = could not find container \"e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3\": container with ID starting with e1520483d3efe45f109163eee12cc7191ca824419b1a17e67b352d77896fd3e3 not found: ID does not exist"
Sep 18 20:39:49.292559 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.292531    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"]
Sep 18 20:39:49.302833 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:49.302810    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2e756-predictor-59c87b9c48-ct7nf"]
Sep 18 20:39:50.223388 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:50.223355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" event={"ID":"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131","Type":"ContainerStarted","Data":"3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155"}
Sep 18 20:39:50.223901 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:50.223426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" event={"ID":"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131","Type":"ContainerStarted","Data":"b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677"}
Sep 18 20:39:50.223901 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:50.223671    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:50.225545 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:50.225520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" event={"ID":"41a06112-ffac-4124-ba1e-ba9ebc52b4d0","Type":"ContainerStarted","Data":"edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727"}
Sep 18 20:39:50.225672 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:50.225576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" event={"ID":"41a06112-ffac-4124-ba1e-ba9ebc52b4d0","Type":"ContainerStarted","Data":"f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09"}
Sep 18 20:39:50.225785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:50.225768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:50.243048 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:50.243009    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podStartSLOduration=2.61871474 podStartE2EDuration="6.242997602s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.624282862s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:39:50.241795251 +0000 UTC m=+1167.051581191" watchObservedRunningTime="2026-09-18 20:39:50.242997602 +0000 UTC m=+1167.052783543"
Sep 18 20:39:50.260800 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:50.260765    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podStartSLOduration=3.28195374 podStartE2EDuration="7.260752615s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.978798875s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:39:50.260079387 +0000 UTC m=+1167.069865330" watchObservedRunningTime="2026-09-18 20:39:50.260752615 +0000 UTC m=+1167.070538556"
Sep 18 20:39:51.235126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:51.235094    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:51.235591 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:51.235187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:51.236088 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:51.236064    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:39:51.236155 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:51.236079    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:39:51.242830 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:51.242799    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" path="/var/lib/kubelet/pods/9848cb9c-0659-4770-9400-35c121b2ed51/volumes"
Sep 18 20:39:52.238566 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:52.238521    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:39:52.239052 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:52.238619    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:39:57.244137 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:57.244103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:39:57.244670 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:57.244643    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:39:57.244747 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:57.244681    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:39:57.245190 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:39:57.245169    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:40:07.245512 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:07.245463    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:40:07.245907 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:07.245463    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:40:17.244954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:17.244917    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:40:17.245500 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:17.245044    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:40:27.244639 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:27.244606    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:40:27.245044 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:27.245024    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:40:37.244885 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:37.244846    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:40:37.245241 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:37.245133    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:40:47.245305 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:47.245259    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:40:47.245783 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:47.245263    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:40:57.244982 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:57.244943    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:40:57.245520 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:57.245080    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:40:58.238099 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:40:58.238061    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 18 20:41:07.246636 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:07.246561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:41:07.379558 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:07.379526    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"]
Sep 18 20:41:07.379793 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:07.379771    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container" containerID="cri-o://b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677" gracePeriod=30
Sep 18 20:41:07.379868 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:07.379838    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kube-rbac-proxy" containerID="cri-o://3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155" gracePeriod=30
Sep 18 20:41:07.380691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:07.380671    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:41:07.511810 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:07.511756    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerID="3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155" exitCode=2
Sep 18 20:41:07.511903 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:07.511826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" event={"ID":"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131","Type":"ContainerDied","Data":"3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155"}
Sep 18 20:41:11.714321 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.714299    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:41:11.846933 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.846909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls\") pod \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") "
Sep 18 20:41:11.847095 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.846948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxtwg\" (UniqueName: \"kubernetes.io/projected/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-api-access-hxtwg\") pod \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") "
Sep 18 20:41:11.847095 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.846979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-rbac-proxy-sar-config\") pod \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") "
Sep 18 20:41:11.847095 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.847036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-cabundle-cert\") pod \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") "
Sep 18 20:41:11.847095 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.847055    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kserve-provision-location\") pod \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\" (UID: \"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131\") "
Sep 18 20:41:11.847418 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.847392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" (UID: "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:11.847514 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.847408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" (UID: "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:41:11.847514 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.847410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" (UID: "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:41:11.849022 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.849002    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" (UID: "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:41:11.849114 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.849096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-api-access-hxtwg" (OuterVolumeSpecName: "kube-api-access-hxtwg") pod "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" (UID: "2ed73cd2-73de-4f8e-8ee7-df2fd81c2131"). InnerVolumeSpecName "kube-api-access-hxtwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:41:11.948309 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.948288    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.948409 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.948311    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.948409 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.948325    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.948409 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.948338    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxtwg\" (UniqueName: \"kubernetes.io/projected/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-api-access-hxtwg\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:11.948409 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:11.948351    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:12.531294 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.531264    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerID="b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677" exitCode=0
Sep 18 20:41:12.531462 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.531319    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"
Sep 18 20:41:12.531462 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.531343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" event={"ID":"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131","Type":"ContainerDied","Data":"b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677"}
Sep 18 20:41:12.531462 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.531402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp" event={"ID":"2ed73cd2-73de-4f8e-8ee7-df2fd81c2131","Type":"ContainerDied","Data":"5125ca833ac1f4f378bb3bac399d43e07b42bfc0a09069e8667363cdfacf96d6"}
Sep 18 20:41:12.531462 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.531428    2577 scope.go:117] "RemoveContainer" containerID="3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155"
Sep 18 20:41:12.539006 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.538983    2577 scope.go:117] "RemoveContainer" containerID="b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677"
Sep 18 20:41:12.545932 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.545914    2577 scope.go:117] "RemoveContainer" containerID="51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903"
Sep 18 20:41:12.551980 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.551959    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"]
Sep 18 20:41:12.553138 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.553125    2577 scope.go:117] "RemoveContainer" containerID="3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155"
Sep 18 20:41:12.553388 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:41:12.553369    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155\": container with ID starting with 3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155 not found: ID does not exist" containerID="3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155"
Sep 18 20:41:12.553435 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.553397    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155"} err="failed to get container status \"3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155\": rpc error: code = NotFound desc = could not find container \"3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155\": container with ID starting with 3f0f69d4d3883b009aa5ff844979bfdb9840caa6fe84c4e76131995443afe155 not found: ID does not exist"
Sep 18 20:41:12.553435 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.553413    2577 scope.go:117] "RemoveContainer" containerID="b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677"
Sep 18 20:41:12.553695 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:41:12.553679    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677\": container with ID starting with b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677 not found: ID does not exist" containerID="b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677"
Sep 18 20:41:12.553739 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.553700    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677"} err="failed to get container status \"b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677\": rpc error: code = NotFound desc = could not find container \"b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677\": container with ID starting with b04f806ecb0e2c8b3cc2c7f898050608aa0e226d596771ae3f8772d7a486f677 not found: ID does not exist"
Sep 18 20:41:12.553739 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.553713    2577 scope.go:117] "RemoveContainer" containerID="51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903"
Sep 18 20:41:12.554014 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:41:12.553989    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903\": container with ID starting with 51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903 not found: ID does not exist" containerID="51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903"
Sep 18 20:41:12.554119 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.554044    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903"} err="failed to get container status \"51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903\": rpc error: code = NotFound desc = could not find container \"51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903\": container with ID starting with 51e6a0e7e55bae6af653e80548defdb0a66f7af3d365f9de8b0aa92c2701b903 not found: ID does not exist"
Sep 18 20:41:12.555561 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:12.555543    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-z62mp"]
Sep 18 20:41:13.242272 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:13.242242    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" path="/var/lib/kubelet/pods/2ed73cd2-73de-4f8e-8ee7-df2fd81c2131/volumes"
Sep 18 20:41:14.025675 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:14.025637    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"]
Sep 18 20:41:14.025918 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:14.025894    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" containerID="cri-o://f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09" gracePeriod=30
Sep 18 20:41:14.026042 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:14.026007    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kube-rbac-proxy" containerID="cri-o://edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727" gracePeriod=30
Sep 18 20:41:14.546583 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:14.546545    2577 generic.go:356] "Generic (PLEG): container finished" podID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerID="edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727" exitCode=2
Sep 18 20:41:14.546583 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:14.546584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" event={"ID":"41a06112-ffac-4124-ba1e-ba9ebc52b4d0","Type":"ContainerDied","Data":"edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727"}
Sep 18 20:41:17.239077 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:17.239038    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 18 20:41:17.245677 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:17.245650    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:41:18.265204 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.265182    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:41:18.390764 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.390683    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-rbac-proxy-sar-config\") pod \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") "
Sep 18 20:41:18.390764 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.390722    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-cabundle-cert\") pod \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") "
Sep 18 20:41:18.390764 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.390750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kserve-provision-location\") pod \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") "
Sep 18 20:41:18.390999 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.390860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-proxy-tls\") pod \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") "
Sep 18 20:41:18.390999 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.390907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vqsk\" (UniqueName: \"kubernetes.io/projected/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-api-access-8vqsk\") pod \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\" (UID: \"41a06112-ffac-4124-ba1e-ba9ebc52b4d0\") "
Sep 18 20:41:18.391107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.391042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41a06112-ffac-4124-ba1e-ba9ebc52b4d0" (UID: "41a06112-ffac-4124-ba1e-ba9ebc52b4d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:41:18.391179 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.391073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41a06112-ffac-4124-ba1e-ba9ebc52b4d0" (UID: "41a06112-ffac-4124-ba1e-ba9ebc52b4d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:41:18.391268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.391247    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41a06112-ffac-4124-ba1e-ba9ebc52b4d0" (UID: "41a06112-ffac-4124-ba1e-ba9ebc52b4d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:41:18.392997 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.392962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-api-access-8vqsk" (OuterVolumeSpecName: "kube-api-access-8vqsk") pod "41a06112-ffac-4124-ba1e-ba9ebc52b4d0" (UID: "41a06112-ffac-4124-ba1e-ba9ebc52b4d0"). InnerVolumeSpecName "kube-api-access-8vqsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:41:18.392997 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.392972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41a06112-ffac-4124-ba1e-ba9ebc52b4d0" (UID: "41a06112-ffac-4124-ba1e-ba9ebc52b4d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:41:18.492032 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.492010    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.492032 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.492032    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.492177 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.492043    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vqsk\" (UniqueName: \"kubernetes.io/projected/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-api-access-8vqsk\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.492177 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.492052    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.492177 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.492062    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41a06112-ffac-4124-ba1e-ba9ebc52b4d0-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:41:18.562093 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.562067    2577 generic.go:356] "Generic (PLEG): container finished" podID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerID="f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09" exitCode=0
Sep 18 20:41:18.562218 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.562109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" event={"ID":"41a06112-ffac-4124-ba1e-ba9ebc52b4d0","Type":"ContainerDied","Data":"f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09"}
Sep 18 20:41:18.562218 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.562133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"
Sep 18 20:41:18.562218 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.562154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7" event={"ID":"41a06112-ffac-4124-ba1e-ba9ebc52b4d0","Type":"ContainerDied","Data":"58f70b73d405aec54ec62860f374b7acbabc2347687db8b8fa58af17cfbcedd5"}
Sep 18 20:41:18.562218 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.562180    2577 scope.go:117] "RemoveContainer" containerID="edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727"
Sep 18 20:41:18.570635 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.570617    2577 scope.go:117] "RemoveContainer" containerID="f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09"
Sep 18 20:41:18.577375 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.577358    2577 scope.go:117] "RemoveContainer" containerID="5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970"
Sep 18 20:41:18.584017 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.583888    2577 scope.go:117] "RemoveContainer" containerID="edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727"
Sep 18 20:41:18.584259 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:41:18.584237    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727\": container with ID starting with edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727 not found: ID does not exist" containerID="edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727"
Sep 18 20:41:18.584360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.584264    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727"} err="failed to get container status \"edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727\": rpc error: code = NotFound desc = could not find container \"edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727\": container with ID starting with edc2db014205f4f02bba048c34d131c07f617422b4e7a83c839cbd73a4dca727 not found: ID does not exist"
Sep 18 20:41:18.584360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.584281    2577 scope.go:117] "RemoveContainer" containerID="f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09"
Sep 18 20:41:18.584587 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:41:18.584560    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09\": container with ID starting with f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09 not found: ID does not exist" containerID="f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09"
Sep 18 20:41:18.584665 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.584594    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09"} err="failed to get container status \"f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09\": rpc error: code = NotFound desc = could not find container \"f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09\": container with ID starting with f815b4ddbc8f29ccdf2766b392ee74539b903b7108942a17ba7d1bf6f9c59e09 not found: ID does not exist"
Sep 18 20:41:18.584665 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.584617    2577 scope.go:117] "RemoveContainer" containerID="5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970"
Sep 18 20:41:18.584900 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:41:18.584883    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970\": container with ID starting with 5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970 not found: ID does not exist" containerID="5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970"
Sep 18 20:41:18.584970 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.584906    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970"} err="failed to get container status \"5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970\": rpc error: code = NotFound desc = could not find container \"5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970\": container with ID starting with 5eb84f0ddbce4e8ba9696f2e951e150499b9adb5b5d1a7b1f3ba259b61ade970 not found: ID does not exist"
Sep 18 20:41:18.585193 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.585175    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"]
Sep 18 20:41:18.595092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:18.595072    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-xfwx7"]
Sep 18 20:41:19.243977 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.243948    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" path="/var/lib/kubelet/pods/41a06112-ffac-4124-ba1e-ba9ebc52b4d0/volumes"
Sep 18 20:41:19.275493 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275446    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"]
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275722    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275734    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275743    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="storage-initializer"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275748    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="storage-initializer"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275756    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kube-rbac-proxy"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275761    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kube-rbac-proxy"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275771    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="storage-initializer"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275777    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="storage-initializer"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275787    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kube-rbac-proxy"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275792    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kube-rbac-proxy"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275805    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275810    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275816    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="storage-initializer"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275821    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="storage-initializer"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275828    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275833    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275839    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kube-rbac-proxy"
Sep 18 20:41:19.275838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275844    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kube-rbac-proxy"
Sep 18 20:41:19.276363 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275881    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kserve-container"
Sep 18 20:41:19.276363 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275888    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kserve-container"
Sep 18 20:41:19.276363 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275894    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed73cd2-73de-4f8e-8ee7-df2fd81c2131" containerName="kube-rbac-proxy"
Sep 18 20:41:19.276363 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275902    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kube-rbac-proxy"
Sep 18 20:41:19.276363 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275908    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a06112-ffac-4124-ba1e-ba9ebc52b4d0" containerName="kube-rbac-proxy"
Sep 18 20:41:19.276363 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.275913    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9848cb9c-0659-4770-9400-35c121b2ed51" containerName="kserve-container"
Sep 18 20:41:19.280271 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.280253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.290830 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.290813    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:41:19.316331 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.316304    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:41:19.317589 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.317560    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 20:41:19.317696 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.317593    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:41:19.317696 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.317623    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:41:19.317696 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.317564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ttgfx\""
Sep 18 20:41:19.317696 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.317564    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:41:19.319117 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.319098    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"]
Sep 18 20:41:19.399099 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.399078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.399199 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.399112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3447a2-120b-4b05-843a-253eba3baf0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.399199 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.399134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.399199 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.399193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cn7z\" (UniqueName: \"kubernetes.io/projected/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-api-access-5cn7z\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.399329 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.399210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3447a2-120b-4b05-843a-253eba3baf0a-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.500459 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.500374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cn7z\" (UniqueName: \"kubernetes.io/projected/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-api-access-5cn7z\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.500459 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.500411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3447a2-120b-4b05-843a-253eba3baf0a-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.500660 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.500537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.500660 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.500580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3447a2-120b-4b05-843a-253eba3baf0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.500660 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.500616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.501070 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.501047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3447a2-120b-4b05-843a-253eba3baf0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.501232 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.501214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.501293 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.501236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.502901 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.502876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3447a2-120b-4b05-843a-253eba3baf0a-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.509805 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.509785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cn7z\" (UniqueName: \"kubernetes.io/projected/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-api-access-5cn7z\") pod \"isvc-canary-promote-predictor-6546f8697f-sgd77\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.590505 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.590463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:19.862126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:19.862096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"]
Sep 18 20:41:19.864599 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:41:19.864573    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3447a2_120b_4b05_843a_253eba3baf0a.slice/crio-e4eadc547e54b269ebbd0fd4bbe108c25479df26527fc949f3dc32b6b99cb56a WatchSource:0}: Error finding container e4eadc547e54b269ebbd0fd4bbe108c25479df26527fc949f3dc32b6b99cb56a: Status 404 returned error can't find the container with id e4eadc547e54b269ebbd0fd4bbe108c25479df26527fc949f3dc32b6b99cb56a
Sep 18 20:41:20.572128 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:20.572086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" event={"ID":"0a3447a2-120b-4b05-843a-253eba3baf0a","Type":"ContainerStarted","Data":"bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07"}
Sep 18 20:41:20.572585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:20.572152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" event={"ID":"0a3447a2-120b-4b05-843a-253eba3baf0a","Type":"ContainerStarted","Data":"e4eadc547e54b269ebbd0fd4bbe108c25479df26527fc949f3dc32b6b99cb56a"}
Sep 18 20:41:24.589977 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:24.589947    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerID="bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07" exitCode=0
Sep 18 20:41:24.590360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:24.590026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" event={"ID":"0a3447a2-120b-4b05-843a-253eba3baf0a","Type":"ContainerDied","Data":"bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07"}
Sep 18 20:41:25.595378 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:25.595344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" event={"ID":"0a3447a2-120b-4b05-843a-253eba3baf0a","Type":"ContainerStarted","Data":"fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d"}
Sep 18 20:41:25.595856 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:25.595411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" event={"ID":"0a3447a2-120b-4b05-843a-253eba3baf0a","Type":"ContainerStarted","Data":"dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7"}
Sep 18 20:41:25.595856 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:25.595646    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:25.625575 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:25.625459    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podStartSLOduration=2.958253936 podStartE2EDuration="6.625445114s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.667191178s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:41:25.624707008 +0000 UTC m=+1262.434492949" watchObservedRunningTime="2026-09-18 20:41:25.625445114 +0000 UTC m=+1262.435231056"
Sep 18 20:41:26.600021 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:26.599992    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:26.601216 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:26.601190    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:41:27.603907 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:27.603871    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:41:32.609434 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:32.609404    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:41:32.610074 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:32.610034    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:41:42.610758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:42.610703    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:41:52.610845 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:41:52.610801    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:42:02.610545 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:02.610504    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:42:12.611004 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:12.610961    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:42:22.610634 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:22.610594    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:42:32.610523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:32.610452    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:42:41.750653 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:41.750568    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"]
Sep 18 20:42:41.751038 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:41.750837    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container" containerID="cri-o://dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7" gracePeriod=30
Sep 18 20:42:41.751038 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:41.750859    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kube-rbac-proxy" containerID="cri-o://fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d" gracePeriod=30
Sep 18 20:42:41.751678 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:41.751618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:42:41.881605 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:41.881566    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerID="fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d" exitCode=2
Sep 18 20:42:41.881769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:41.881665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" event={"ID":"0a3447a2-120b-4b05-843a-253eba3baf0a","Type":"ContainerDied","Data":"fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d"}
Sep 18 20:42:42.604656 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:42.604610    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 18 20:42:46.482973 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.482952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:42:46.566730 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.566670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cn7z\" (UniqueName: \"kubernetes.io/projected/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-api-access-5cn7z\") pod \"0a3447a2-120b-4b05-843a-253eba3baf0a\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") "
Sep 18 20:42:46.566730 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.566704    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-rbac-proxy-sar-config\") pod \"0a3447a2-120b-4b05-843a-253eba3baf0a\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") "
Sep 18 20:42:46.566886 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.566748    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3447a2-120b-4b05-843a-253eba3baf0a-kserve-provision-location\") pod \"0a3447a2-120b-4b05-843a-253eba3baf0a\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") "
Sep 18 20:42:46.566886 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.566780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-cabundle-cert\") pod \"0a3447a2-120b-4b05-843a-253eba3baf0a\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") "
Sep 18 20:42:46.566886 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.566834    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3447a2-120b-4b05-843a-253eba3baf0a-proxy-tls\") pod \"0a3447a2-120b-4b05-843a-253eba3baf0a\" (UID: \"0a3447a2-120b-4b05-843a-253eba3baf0a\") "
Sep 18 20:42:46.567093 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.567070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a3447a2-120b-4b05-843a-253eba3baf0a" (UID: "0a3447a2-120b-4b05-843a-253eba3baf0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:42:46.567160 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.567098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3447a2-120b-4b05-843a-253eba3baf0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a3447a2-120b-4b05-843a-253eba3baf0a" (UID: "0a3447a2-120b-4b05-843a-253eba3baf0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:42:46.567160 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.567138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a3447a2-120b-4b05-843a-253eba3baf0a" (UID: "0a3447a2-120b-4b05-843a-253eba3baf0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:42:46.568761 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.568735    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-api-access-5cn7z" (OuterVolumeSpecName: "kube-api-access-5cn7z") pod "0a3447a2-120b-4b05-843a-253eba3baf0a" (UID: "0a3447a2-120b-4b05-843a-253eba3baf0a"). InnerVolumeSpecName "kube-api-access-5cn7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:42:46.568847 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.568773    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3447a2-120b-4b05-843a-253eba3baf0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a3447a2-120b-4b05-843a-253eba3baf0a" (UID: "0a3447a2-120b-4b05-843a-253eba3baf0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:42:46.667357 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.667332    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cn7z\" (UniqueName: \"kubernetes.io/projected/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-api-access-5cn7z\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:42:46.667357 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.667356    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:42:46.667523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.667365    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3447a2-120b-4b05-843a-253eba3baf0a-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:42:46.667523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.667373    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3447a2-120b-4b05-843a-253eba3baf0a-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:42:46.667523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.667382    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3447a2-120b-4b05-843a-253eba3baf0a-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:42:46.904154 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.904124    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerID="dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7" exitCode=0
Sep 18 20:42:46.904308 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.904175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" event={"ID":"0a3447a2-120b-4b05-843a-253eba3baf0a","Type":"ContainerDied","Data":"dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7"}
Sep 18 20:42:46.904308 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.904236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77" event={"ID":"0a3447a2-120b-4b05-843a-253eba3baf0a","Type":"ContainerDied","Data":"e4eadc547e54b269ebbd0fd4bbe108c25479df26527fc949f3dc32b6b99cb56a"}
Sep 18 20:42:46.904308 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.904264    2577 scope.go:117] "RemoveContainer" containerID="fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d"
Sep 18 20:42:46.904501 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.904184    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"
Sep 18 20:42:46.912428 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.912411    2577 scope.go:117] "RemoveContainer" containerID="dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7"
Sep 18 20:42:46.925114 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.925095    2577 scope.go:117] "RemoveContainer" containerID="bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07"
Sep 18 20:42:46.928403 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.928371    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"]
Sep 18 20:42:46.932097 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.932075    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-sgd77"]
Sep 18 20:42:46.932750 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.932735    2577 scope.go:117] "RemoveContainer" containerID="fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d"
Sep 18 20:42:46.932998 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:42:46.932979    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d\": container with ID starting with fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d not found: ID does not exist" containerID="fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d"
Sep 18 20:42:46.933053 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.933005    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d"} err="failed to get container status \"fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d\": rpc error: code = NotFound desc = could not find container \"fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d\": container with ID starting with fe38d2af55e094d842fed3a04065c1d8722c4d498e094fb14fd69d180c856c8d not found: ID does not exist"
Sep 18 20:42:46.933053 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.933022    2577 scope.go:117] "RemoveContainer" containerID="dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7"
Sep 18 20:42:46.933284 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:42:46.933266    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7\": container with ID starting with dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7 not found: ID does not exist" containerID="dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7"
Sep 18 20:42:46.933339 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.933290    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7"} err="failed to get container status \"dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7\": rpc error: code = NotFound desc = could not find container \"dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7\": container with ID starting with dd1901f4759e8480f1dfa37d2ad8eafbcc991f92e51bfdd3494fec7000453aa7 not found: ID does not exist"
Sep 18 20:42:46.933339 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.933305    2577 scope.go:117] "RemoveContainer" containerID="bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07"
Sep 18 20:42:46.933556 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:42:46.933539    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07\": container with ID starting with bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07 not found: ID does not exist" containerID="bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07"
Sep 18 20:42:46.933601 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:46.933560    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07"} err="failed to get container status \"bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07\": rpc error: code = NotFound desc = could not find container \"bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07\": container with ID starting with bf257349644834dd21b93d6ff0fb08ebbfb2a3f8c7bcaa7fdad853b4c82e0e07 not found: ID does not exist"
Sep 18 20:42:47.242651 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:47.242580    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" path="/var/lib/kubelet/pods/0a3447a2-120b-4b05-843a-253eba3baf0a/volumes"
Sep 18 20:42:49.379089 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379061    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"]
Sep 18 20:42:49.379436 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379356    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kube-rbac-proxy"
Sep 18 20:42:49.379436 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379367    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kube-rbac-proxy"
Sep 18 20:42:49.379436 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379384    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="storage-initializer"
Sep 18 20:42:49.379436 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379389    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="storage-initializer"
Sep 18 20:42:49.379436 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379395    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container"
Sep 18 20:42:49.379436 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379400    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container"
Sep 18 20:42:49.379658 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379446    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kube-rbac-proxy"
Sep 18 20:42:49.379658 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.379455    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3447a2-120b-4b05-843a-253eba3baf0a" containerName="kserve-container"
Sep 18 20:42:49.383815 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.383798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.386702 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.386682    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:42:49.386820 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.386681    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:42:49.387672 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.387656    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:42:49.387787 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.387673    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 20:42:49.387787 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.387710    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ttgfx\""
Sep 18 20:42:49.387787 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.387735    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:42:49.387925 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.387789    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:42:49.396045 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.396025    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"]
Sep 18 20:42:49.487045 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.487011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.487045 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.487049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9af7e58a-12e1-454e-a386-d42946605389-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.487257 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.487082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph75n\" (UniqueName: \"kubernetes.io/projected/9af7e58a-12e1-454e-a386-d42946605389-kube-api-access-ph75n\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.487257 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.487127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.487257 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.487159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af7e58a-12e1-454e-a386-d42946605389-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.588400 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.588364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af7e58a-12e1-454e-a386-d42946605389-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.588573 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.588413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.588573 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.588433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9af7e58a-12e1-454e-a386-d42946605389-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.588573 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.588465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph75n\" (UniqueName: \"kubernetes.io/projected/9af7e58a-12e1-454e-a386-d42946605389-kube-api-access-ph75n\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.588573 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.588503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.588904 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.588877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af7e58a-12e1-454e-a386-d42946605389-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.589161 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.589141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.589226 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.589149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.590845 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.590821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9af7e58a-12e1-454e-a386-d42946605389-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.597302 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.597282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph75n\" (UniqueName: \"kubernetes.io/projected/9af7e58a-12e1-454e-a386-d42946605389-kube-api-access-ph75n\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.693448 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.693394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:49.963855 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:49.963831    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"]
Sep 18 20:42:49.965840 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:42:49.965812    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af7e58a_12e1_454e_a386_d42946605389.slice/crio-405a9ab26d0d24e9e72325652cb453bf4bb74d6a93ed349dfbd7c1d4eae515fd WatchSource:0}: Error finding container 405a9ab26d0d24e9e72325652cb453bf4bb74d6a93ed349dfbd7c1d4eae515fd: Status 404 returned error can't find the container with id 405a9ab26d0d24e9e72325652cb453bf4bb74d6a93ed349dfbd7c1d4eae515fd
Sep 18 20:42:50.922949 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:50.922916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" event={"ID":"9af7e58a-12e1-454e-a386-d42946605389","Type":"ContainerStarted","Data":"8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697"}
Sep 18 20:42:50.923313 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:50.922976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" event={"ID":"9af7e58a-12e1-454e-a386-d42946605389","Type":"ContainerStarted","Data":"405a9ab26d0d24e9e72325652cb453bf4bb74d6a93ed349dfbd7c1d4eae515fd"}
Sep 18 20:42:53.935316 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:53.935284    2577 generic.go:356] "Generic (PLEG): container finished" podID="9af7e58a-12e1-454e-a386-d42946605389" containerID="8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697" exitCode=0
Sep 18 20:42:53.935638 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:53.935359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" event={"ID":"9af7e58a-12e1-454e-a386-d42946605389","Type":"ContainerDied","Data":"8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697"}
Sep 18 20:42:54.940545 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:54.940513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" event={"ID":"9af7e58a-12e1-454e-a386-d42946605389","Type":"ContainerStarted","Data":"5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d"}
Sep 18 20:42:54.940900 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:54.940575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" event={"ID":"9af7e58a-12e1-454e-a386-d42946605389","Type":"ContainerStarted","Data":"6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af"}
Sep 18 20:42:54.940900 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:54.940775    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:54.963184 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:54.963139    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podStartSLOduration=2.093078942 podStartE2EDuration="5.963125337s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.870046395s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:42:54.961668501 +0000 UTC m=+1351.771454467" watchObservedRunningTime="2026-09-18 20:42:54.963125337 +0000 UTC m=+1351.772911443"
Sep 18 20:42:55.944595 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:55.944561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:42:55.945742 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:55.945714    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:42:56.947714 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:42:56.947676    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:43:01.952070 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:43:01.952042    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:43:01.952597 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:43:01.952568    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:43:11.953402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:43:11.953363    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:43:21.956129 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:43:21.956073    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:43:31.953064 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:43:31.953023    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:43:41.952729 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:43:41.952685    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:43:51.953164 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:43:51.953122    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:44:01.953327 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:01.953284    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:44:11.953922 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:11.953847    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:44:19.507857 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:19.507825    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"]
Sep 18 20:44:19.508288 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:19.508078    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" containerID="cri-o://6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af" gracePeriod=30
Sep 18 20:44:19.508288 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:19.508109    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kube-rbac-proxy" containerID="cri-o://5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d" gracePeriod=30
Sep 18 20:44:20.264336 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:20.264305    2577 generic.go:356] "Generic (PLEG): container finished" podID="9af7e58a-12e1-454e-a386-d42946605389" containerID="5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d" exitCode=2
Sep 18 20:44:20.264336 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:20.264341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" event={"ID":"9af7e58a-12e1-454e-a386-d42946605389","Type":"ContainerDied","Data":"5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d"}
Sep 18 20:44:21.948800 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:21.948760    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 18 20:44:21.953158 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:21.953126    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:44:24.150961 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.150939    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:44:24.205209 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-cabundle-cert\") pod \"9af7e58a-12e1-454e-a386-d42946605389\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") "
Sep 18 20:44:24.205209 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph75n\" (UniqueName: \"kubernetes.io/projected/9af7e58a-12e1-454e-a386-d42946605389-kube-api-access-ph75n\") pod \"9af7e58a-12e1-454e-a386-d42946605389\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") "
Sep 18 20:44:24.205358 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af7e58a-12e1-454e-a386-d42946605389-kserve-provision-location\") pod \"9af7e58a-12e1-454e-a386-d42946605389\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") "
Sep 18 20:44:24.205410 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205349    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-kube-rbac-proxy-sar-config\") pod \"9af7e58a-12e1-454e-a386-d42946605389\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") "
Sep 18 20:44:24.205410 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9af7e58a-12e1-454e-a386-d42946605389-proxy-tls\") pod \"9af7e58a-12e1-454e-a386-d42946605389\" (UID: \"9af7e58a-12e1-454e-a386-d42946605389\") "
Sep 18 20:44:24.205539 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af7e58a-12e1-454e-a386-d42946605389-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9af7e58a-12e1-454e-a386-d42946605389" (UID: "9af7e58a-12e1-454e-a386-d42946605389"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:24.205539 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205517    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9af7e58a-12e1-454e-a386-d42946605389" (UID: "9af7e58a-12e1-454e-a386-d42946605389"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:44:24.205661 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205645    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.205724 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205662    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9af7e58a-12e1-454e-a386-d42946605389-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.205775 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.205732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9af7e58a-12e1-454e-a386-d42946605389" (UID: "9af7e58a-12e1-454e-a386-d42946605389"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:44:24.207265 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.207241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af7e58a-12e1-454e-a386-d42946605389-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9af7e58a-12e1-454e-a386-d42946605389" (UID: "9af7e58a-12e1-454e-a386-d42946605389"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:44:24.207382 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.207323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af7e58a-12e1-454e-a386-d42946605389-kube-api-access-ph75n" (OuterVolumeSpecName: "kube-api-access-ph75n") pod "9af7e58a-12e1-454e-a386-d42946605389" (UID: "9af7e58a-12e1-454e-a386-d42946605389"). InnerVolumeSpecName "kube-api-access-ph75n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:44:24.281835 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.281810    2577 generic.go:356] "Generic (PLEG): container finished" podID="9af7e58a-12e1-454e-a386-d42946605389" containerID="6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af" exitCode=0
Sep 18 20:44:24.281954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.281864    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"
Sep 18 20:44:24.281954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.281885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" event={"ID":"9af7e58a-12e1-454e-a386-d42946605389","Type":"ContainerDied","Data":"6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af"}
Sep 18 20:44:24.281954 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.281938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl" event={"ID":"9af7e58a-12e1-454e-a386-d42946605389","Type":"ContainerDied","Data":"405a9ab26d0d24e9e72325652cb453bf4bb74d6a93ed349dfbd7c1d4eae515fd"}
Sep 18 20:44:24.282121 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.281963    2577 scope.go:117] "RemoveContainer" containerID="5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d"
Sep 18 20:44:24.290126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.290094    2577 scope.go:117] "RemoveContainer" containerID="6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af"
Sep 18 20:44:24.297251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.297235    2577 scope.go:117] "RemoveContainer" containerID="8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697"
Sep 18 20:44:24.303190 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.303166    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"]
Sep 18 20:44:24.304535 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.304522    2577 scope.go:117] "RemoveContainer" containerID="5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d"
Sep 18 20:44:24.304809 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:44:24.304781    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d\": container with ID starting with 5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d not found: ID does not exist" containerID="5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d"
Sep 18 20:44:24.304858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.304810    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d"} err="failed to get container status \"5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d\": rpc error: code = NotFound desc = could not find container \"5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d\": container with ID starting with 5d295d1a4ee39bc662583f77926fe6c9f021c92340e3f726172ffcc0f296f25d not found: ID does not exist"
Sep 18 20:44:24.304858 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.304826    2577 scope.go:117] "RemoveContainer" containerID="6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af"
Sep 18 20:44:24.305081 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:44:24.305065    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af\": container with ID starting with 6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af not found: ID does not exist" containerID="6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af"
Sep 18 20:44:24.305148 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.305103    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af"} err="failed to get container status \"6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af\": rpc error: code = NotFound desc = could not find container \"6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af\": container with ID starting with 6a174c073fe62e62e68048fbabf6211977e4d3bbb5f66092144fe69d987d65af not found: ID does not exist"
Sep 18 20:44:24.305148 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.305124    2577 scope.go:117] "RemoveContainer" containerID="8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697"
Sep 18 20:44:24.305385 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:44:24.305367    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697\": container with ID starting with 8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697 not found: ID does not exist" containerID="8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697"
Sep 18 20:44:24.305434 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.305390    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697"} err="failed to get container status \"8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697\": rpc error: code = NotFound desc = could not find container \"8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697\": container with ID starting with 8a49e3499ced4a374b32237452d9cfb5b8f392f9aadb0425e2f8945e79847697 not found: ID does not exist"
Sep 18 20:44:24.306723 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.306698    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9af7e58a-12e1-454e-a386-d42946605389-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.306723 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.306725    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph75n\" (UniqueName: \"kubernetes.io/projected/9af7e58a-12e1-454e-a386-d42946605389-kube-api-access-ph75n\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.306842 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.306735    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9af7e58a-12e1-454e-a386-d42946605389-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:44:24.308754 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:24.308736    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-qv2jl"]
Sep 18 20:44:25.242384 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:25.242350    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9af7e58a-12e1-454e-a386-d42946605389" path="/var/lib/kubelet/pods/9af7e58a-12e1-454e-a386-d42946605389/volumes"
Sep 18 20:44:34.000957 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.000925    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"]
Sep 18 20:44:34.001402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.001195    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container"
Sep 18 20:44:34.001402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.001206    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container"
Sep 18 20:44:34.001402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.001218    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="storage-initializer"
Sep 18 20:44:34.001402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.001223    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="storage-initializer"
Sep 18 20:44:34.001402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.001230    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kube-rbac-proxy"
Sep 18 20:44:34.001402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.001236    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kube-rbac-proxy"
Sep 18 20:44:34.001402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.001279    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kube-rbac-proxy"
Sep 18 20:44:34.001402 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.001287    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af7e58a-12e1-454e-a386-d42946605389" containerName="kserve-container"
Sep 18 20:44:34.005648 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.005631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.008244 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.008225    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:44:34.008349 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.008230    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 18 20:44:34.009470 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.009448    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:44:34.009583 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.009455    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 18 20:44:34.009643 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.009588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ttgfx\""
Sep 18 20:44:34.014191 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.014168    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"]
Sep 18 20:44:34.083117 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.083077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52ff\" (UniqueName: \"kubernetes.io/projected/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-api-access-t52ff\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.083268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.083131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f63558-cd70-4436-8229-e1d2e3eca771-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.083268 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.083206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.184395 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.184362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52ff\" (UniqueName: \"kubernetes.io/projected/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-api-access-t52ff\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.184537 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.184416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f63558-cd70-4436-8229-e1d2e3eca771-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.184537 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.184451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.185107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.185085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.186845 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.186827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f63558-cd70-4436-8229-e1d2e3eca771-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.193193 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.193174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52ff\" (UniqueName: \"kubernetes.io/projected/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-api-access-t52ff\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xlvn9\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.316196 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.316132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:34.587709 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.587682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"]
Sep 18 20:44:34.589638 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:44:34.589597    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f63558_cd70_4436_8229_e1d2e3eca771.slice/crio-6a031b635c0cf872ddfd37d9db5450180dec4e52907e29b748c79260623b5813 WatchSource:0}: Error finding container 6a031b635c0cf872ddfd37d9db5450180dec4e52907e29b748c79260623b5813: Status 404 returned error can't find the container with id 6a031b635c0cf872ddfd37d9db5450180dec4e52907e29b748c79260623b5813
Sep 18 20:44:34.591439 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:34.591422    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:44:35.325666 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:35.325623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" event={"ID":"f8f63558-cd70-4436-8229-e1d2e3eca771","Type":"ContainerStarted","Data":"6a031b635c0cf872ddfd37d9db5450180dec4e52907e29b748c79260623b5813"}
Sep 18 20:44:36.331186 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:36.331154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" event={"ID":"f8f63558-cd70-4436-8229-e1d2e3eca771","Type":"ContainerStarted","Data":"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d"}
Sep 18 20:44:36.331593 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:36.331209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" event={"ID":"f8f63558-cd70-4436-8229-e1d2e3eca771","Type":"ContainerStarted","Data":"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b"}
Sep 18 20:44:36.331593 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:36.331266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:36.351775 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:36.351722    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" podStartSLOduration=2.308128222 podStartE2EDuration="3.351707953s" totalImagesPullingTime="1.043579731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:44:36.350394232 +0000 UTC m=+1453.160180383" watchObservedRunningTime="2026-09-18 20:44:36.351707953 +0000 UTC m=+1453.161493948"
Sep 18 20:44:37.334559 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:37.334524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:37.336395 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:37.336374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:44.346020 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:44.345990    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:44:54.040872 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.040833    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"]
Sep 18 20:44:54.044602 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.044579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.047056 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.047031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:44:54.047056 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.047034    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 18 20:44:54.047220 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.047041    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:44:54.047220 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.047042    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 18 20:44:54.058059 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.058039    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"]
Sep 18 20:44:54.130338 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.130317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b1feec-b214-4317-b226-fe41a312b38c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.130463 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.130345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.130463 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.130370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.130463 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.130429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98d9c\" (UniqueName: \"kubernetes.io/projected/19b1feec-b214-4317-b226-fe41a312b38c-kube-api-access-98d9c\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.130654 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.130508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.230895 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.230867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b1feec-b214-4317-b226-fe41a312b38c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.231016 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.230900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.231016 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.230936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.231126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.231038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98d9c\" (UniqueName: \"kubernetes.io/projected/19b1feec-b214-4317-b226-fe41a312b38c-kube-api-access-98d9c\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.231126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.231085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.231227 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:44:54.231201    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 18 20:44:54.231284 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.231253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b1feec-b214-4317-b226-fe41a312b38c-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.231284 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:44:54.231279    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls podName:19b1feec-b214-4317-b226-fe41a312b38c nodeName:}" failed. No retries permitted until 2026-09-18 20:44:54.731258045 +0000 UTC m=+1471.541043972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls") pod "isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" (UID: "19b1feec-b214-4317-b226-fe41a312b38c") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 18 20:44:54.231622 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.231602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.231710 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.231662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.241523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.241505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98d9c\" (UniqueName: \"kubernetes.io/projected/19b1feec-b214-4317-b226-fe41a312b38c-kube-api-access-98d9c\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.736087 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.736054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.738257 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.738231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-h5cf7\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:54.957358 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:54.957328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:44:55.232251 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:55.232227    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"]
Sep 18 20:44:55.234381 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:44:55.234351    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b1feec_b214_4317_b226_fe41a312b38c.slice/crio-0acb5bf5d094a39548d6a1651e0cecb080e7d043f5f6ef7e2b11eae6ca5125d1 WatchSource:0}: Error finding container 0acb5bf5d094a39548d6a1651e0cecb080e7d043f5f6ef7e2b11eae6ca5125d1: Status 404 returned error can't find the container with id 0acb5bf5d094a39548d6a1651e0cecb080e7d043f5f6ef7e2b11eae6ca5125d1
Sep 18 20:44:55.401201 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:55.401160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerStarted","Data":"e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff"}
Sep 18 20:44:55.401379 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:55.401236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerStarted","Data":"0acb5bf5d094a39548d6a1651e0cecb080e7d043f5f6ef7e2b11eae6ca5125d1"}
Sep 18 20:44:59.417330 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:59.417300    2577 generic.go:356] "Generic (PLEG): container finished" podID="19b1feec-b214-4317-b226-fe41a312b38c" containerID="e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff" exitCode=0
Sep 18 20:44:59.417703 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:44:59.417378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerDied","Data":"e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff"}
Sep 18 20:45:00.423832 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:00.423798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerStarted","Data":"24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc"}
Sep 18 20:45:00.424207 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:00.423866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerStarted","Data":"29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920"}
Sep 18 20:45:00.424207 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:00.423891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerStarted","Data":"f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8"}
Sep 18 20:45:00.424207 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:00.424161    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:45:00.424367 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:00.424212    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:45:00.425455 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:00.425423    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:00.443639 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:00.443589    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podStartSLOduration=2.442060945 podStartE2EDuration="6.443576882s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.001515937s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:45:00.442036138 +0000 UTC m=+1477.251822081" watchObservedRunningTime="2026-09-18 20:45:00.443576882 +0000 UTC m=+1477.253362825"
Sep 18 20:45:01.427968 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:01.427909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:45:01.428411 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:01.428116    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:01.429222 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:01.429197    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:45:02.431599 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:02.431559    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:02.432016 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:02.431906    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:45:07.436046 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:07.436019    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:45:07.436656 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:07.436620    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:07.437076 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:07.437053    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:45:17.436795 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:17.436753    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:17.437335 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:17.437306    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:45:27.437124 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:27.437088    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:27.437566 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:27.437543    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:45:37.436770 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:37.436674    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:37.437174 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:37.437065    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:45:47.437435 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:47.437389    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:47.437992 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:47.437830    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:45:57.436662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:57.436612    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:45:57.437096 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:45:57.436993    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:46:07.436881 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:07.436841    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:46:07.439177 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:07.437264    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:46:10.238653 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:10.238609    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:46:10.239085 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:10.239017    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:46:20.240191 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:20.240160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:46:20.240668 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:20.240292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:46:29.488183 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.488132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-xlvn9_f8f63558-cd70-4436-8229-e1d2e3eca771/kserve-container/0.log"
Sep 18 20:46:29.690910 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.690880    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"]
Sep 18 20:46:29.691185 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.691153    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy" containerID="cri-o://29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920" gracePeriod=30
Sep 18 20:46:29.691266 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.691149    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" containerID="cri-o://24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc" gracePeriod=30
Sep 18 20:46:29.691266 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.691145    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" containerID="cri-o://f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8" gracePeriod=30
Sep 18 20:46:29.733823 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.733796    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"]
Sep 18 20:46:29.734043 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.734007    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerName="kserve-container" containerID="cri-o://bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b" gracePeriod=30
Sep 18 20:46:29.734102 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.734027    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerName="kube-rbac-proxy" containerID="cri-o://e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d" gracePeriod=30
Sep 18 20:46:29.973882 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:29.973860    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:46:30.112769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.112741    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-rbac-proxy-sar-config\") pod \"f8f63558-cd70-4436-8229-e1d2e3eca771\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") "
Sep 18 20:46:30.112920 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.112777    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f63558-cd70-4436-8229-e1d2e3eca771-proxy-tls\") pod \"f8f63558-cd70-4436-8229-e1d2e3eca771\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") "
Sep 18 20:46:30.112920 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.112907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t52ff\" (UniqueName: \"kubernetes.io/projected/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-api-access-t52ff\") pod \"f8f63558-cd70-4436-8229-e1d2e3eca771\" (UID: \"f8f63558-cd70-4436-8229-e1d2e3eca771\") "
Sep 18 20:46:30.113164 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.113139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8f63558-cd70-4436-8229-e1d2e3eca771" (UID: "f8f63558-cd70-4436-8229-e1d2e3eca771"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:46:30.114878 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.114849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-api-access-t52ff" (OuterVolumeSpecName: "kube-api-access-t52ff") pod "f8f63558-cd70-4436-8229-e1d2e3eca771" (UID: "f8f63558-cd70-4436-8229-e1d2e3eca771"). InnerVolumeSpecName "kube-api-access-t52ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:46:30.114969 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.114877    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f63558-cd70-4436-8229-e1d2e3eca771-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8f63558-cd70-4436-8229-e1d2e3eca771" (UID: "f8f63558-cd70-4436-8229-e1d2e3eca771"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:46:30.214357 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.214332    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52ff\" (UniqueName: \"kubernetes.io/projected/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-api-access-t52ff\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.214357 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.214355    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f63558-cd70-4436-8229-e1d2e3eca771-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.214545 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.214365    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f63558-cd70-4436-8229-e1d2e3eca771-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.238590 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.238552    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:46:30.238920 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.238893    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:46:30.764143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.764114    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerID="e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d" exitCode=2
Sep 18 20:46:30.764143 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.764134    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerID="bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b" exitCode=2
Sep 18 20:46:30.764640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.764171    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"
Sep 18 20:46:30.764640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.764189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" event={"ID":"f8f63558-cd70-4436-8229-e1d2e3eca771","Type":"ContainerDied","Data":"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d"}
Sep 18 20:46:30.764640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.764240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" event={"ID":"f8f63558-cd70-4436-8229-e1d2e3eca771","Type":"ContainerDied","Data":"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b"}
Sep 18 20:46:30.764640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.764273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9" event={"ID":"f8f63558-cd70-4436-8229-e1d2e3eca771","Type":"ContainerDied","Data":"6a031b635c0cf872ddfd37d9db5450180dec4e52907e29b748c79260623b5813"}
Sep 18 20:46:30.764640 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.764303    2577 scope.go:117] "RemoveContainer" containerID="e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d"
Sep 18 20:46:30.766789 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.766763    2577 generic.go:356] "Generic (PLEG): container finished" podID="19b1feec-b214-4317-b226-fe41a312b38c" containerID="29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920" exitCode=2
Sep 18 20:46:30.766891 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.766829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerDied","Data":"29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920"}
Sep 18 20:46:30.772514 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.772497    2577 scope.go:117] "RemoveContainer" containerID="bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b"
Sep 18 20:46:30.779527 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.779507    2577 scope.go:117] "RemoveContainer" containerID="e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d"
Sep 18 20:46:30.779903 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:46:30.779886    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d\": container with ID starting with e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d not found: ID does not exist" containerID="e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d"
Sep 18 20:46:30.779970 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.779908    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d"} err="failed to get container status \"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d\": rpc error: code = NotFound desc = could not find container \"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d\": container with ID starting with e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d not found: ID does not exist"
Sep 18 20:46:30.779970 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.779922    2577 scope.go:117] "RemoveContainer" containerID="bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b"
Sep 18 20:46:30.780195 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:46:30.780177    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b\": container with ID starting with bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b not found: ID does not exist" containerID="bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b"
Sep 18 20:46:30.780242 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.780201    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b"} err="failed to get container status \"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b\": rpc error: code = NotFound desc = could not find container \"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b\": container with ID starting with bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b not found: ID does not exist"
Sep 18 20:46:30.780242 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.780217    2577 scope.go:117] "RemoveContainer" containerID="e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d"
Sep 18 20:46:30.780460 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.780442    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d"} err="failed to get container status \"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d\": rpc error: code = NotFound desc = could not find container \"e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d\": container with ID starting with e88cee990c0512bd68774741b958148a57a59937f9ca95c263181f453f35144d not found: ID does not exist"
Sep 18 20:46:30.780517 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.780461    2577 scope.go:117] "RemoveContainer" containerID="bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b"
Sep 18 20:46:30.780700 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.780682    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b"} err="failed to get container status \"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b\": rpc error: code = NotFound desc = could not find container \"bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b\": container with ID starting with bb0ab219efd32c2ec6ceb3d484000661c60ae9535f521e82d60ed2091c44dd2b not found: ID does not exist"
Sep 18 20:46:30.784768 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.784748    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"]
Sep 18 20:46:30.788221 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:30.788203    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xlvn9"]
Sep 18 20:46:31.054655 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.054551    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4qczb/must-gather-fh7b9"]
Sep 18 20:46:31.055237 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.055214    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerName="kserve-container"
Sep 18 20:46:31.055237 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.055239    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerName="kserve-container"
Sep 18 20:46:31.055407 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.055258    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerName="kube-rbac-proxy"
Sep 18 20:46:31.055407 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.055268    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerName="kube-rbac-proxy"
Sep 18 20:46:31.055539 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.055419    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerName="kserve-container"
Sep 18 20:46:31.055539 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.055433    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" containerName="kube-rbac-proxy"
Sep 18 20:46:31.060308 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.060289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:46:31.062927 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.062908    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qczb\"/\"kube-root-ca.crt\""
Sep 18 20:46:31.063175 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.063158    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4qczb\"/\"default-dockercfg-mg7b8\""
Sep 18 20:46:31.063261 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.063162    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qczb\"/\"openshift-service-ca.crt\""
Sep 18 20:46:31.068091 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.068069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qczb/must-gather-fh7b9"]
Sep 18 20:46:31.119877 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.119858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-must-gather-output\") pod \"must-gather-fh7b9\" (UID: \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\") " pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:46:31.119974 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.119887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-868vk\" (UniqueName: \"kubernetes.io/projected/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-kube-api-access-868vk\") pod \"must-gather-fh7b9\" (UID: \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\") " pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:46:31.220823 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.220799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-must-gather-output\") pod \"must-gather-fh7b9\" (UID: \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\") " pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:46:31.220951 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.220829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-868vk\" (UniqueName: \"kubernetes.io/projected/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-kube-api-access-868vk\") pod \"must-gather-fh7b9\" (UID: \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\") " pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:46:31.221107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.221089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-must-gather-output\") pod \"must-gather-fh7b9\" (UID: \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\") " pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:46:31.228875 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.228849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-868vk\" (UniqueName: \"kubernetes.io/projected/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-kube-api-access-868vk\") pod \"must-gather-fh7b9\" (UID: \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\") " pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:46:31.242461 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.242436    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f63558-cd70-4436-8229-e1d2e3eca771" path="/var/lib/kubelet/pods/f8f63558-cd70-4436-8229-e1d2e3eca771/volumes"
Sep 18 20:46:31.380895 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.380864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:46:31.647925 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.647901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qczb/must-gather-fh7b9"]
Sep 18 20:46:31.649983 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:46:31.649954    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod688dbfb2_d3b6_4c79_8fb7_51f05569ed9b.slice/crio-d7a8816556df57bfdbd7e25039fff1139b14bfdb44e0da3137ea42faa7ca7ad9 WatchSource:0}: Error finding container d7a8816556df57bfdbd7e25039fff1139b14bfdb44e0da3137ea42faa7ca7ad9: Status 404 returned error can't find the container with id d7a8816556df57bfdbd7e25039fff1139b14bfdb44e0da3137ea42faa7ca7ad9
Sep 18 20:46:31.773126 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:31.773087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qczb/must-gather-fh7b9" event={"ID":"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b","Type":"ContainerStarted","Data":"d7a8816556df57bfdbd7e25039fff1139b14bfdb44e0da3137ea42faa7ca7ad9"}
Sep 18 20:46:32.433502 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:32.433430    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 18 20:46:35.799424 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:35.799342    2577 generic.go:356] "Generic (PLEG): container finished" podID="19b1feec-b214-4317-b226-fe41a312b38c" containerID="f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8" exitCode=0
Sep 18 20:46:35.799424 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:35.799400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerDied","Data":"f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8"}
Sep 18 20:46:37.431861 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:37.431820    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 18 20:46:38.818688 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:38.818653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qczb/must-gather-fh7b9" event={"ID":"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b","Type":"ContainerStarted","Data":"cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43"}
Sep 18 20:46:38.819050 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:38.818697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qczb/must-gather-fh7b9" event={"ID":"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b","Type":"ContainerStarted","Data":"ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b"}
Sep 18 20:46:38.835253 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:38.835203    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4qczb/must-gather-fh7b9" podStartSLOduration=1.7617735209999998 podStartE2EDuration="7.835188019s" totalImagesPullingTime="6.073414498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:46:38.833798566 +0000 UTC m=+1575.643584508" watchObservedRunningTime="2026-09-18 20:46:38.835188019 +0000 UTC m=+1575.644973962"
Sep 18 20:46:40.238793 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:40.238750    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:46:40.239204 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:40.239134    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:46:42.432015 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:42.431968    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 18 20:46:42.432423 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:42.432129    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:46:46.608331 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:46.608302    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:46.634027 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:46.633990    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:46.645813 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:46.645783    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:46.658273 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:46.658246    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:47.432589 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:47.432551    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 18 20:46:47.504765 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:47.504740    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:47.518496 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:47.518453    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:47.529591 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:47.529567    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:47.540873 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:47.540847    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:48.371957 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:48.371925    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:48.386599 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:48.386567    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:48.400294 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:48.400270    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:48.412747 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:48.412727    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:49.241645 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:49.241613    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:49.254613 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:49.254583    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:49.266159 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:49.266136    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:49.278841 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:49.278804    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:50.111156 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.111126    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:50.125610 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.125575    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:50.138317 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.138299    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:50.150579 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.150547    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:50.239288 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.239248    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:46:50.239458 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.239436    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:46:50.239585 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.239559    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:46:50.239697 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.239684    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:46:50.960254 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.960217    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:50.972785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.972753    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:50.983620 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.983600    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:50.994937 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:50.994919    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:51.802757 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:51.802728    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:51.815729 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:51.815705    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:51.826101 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:51.826082    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:51.837685 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:51.837661    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:52.432131 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:52.432084    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 18 20:46:52.648840 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:52.648807    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:52.660433 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:52.660407    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:52.670525 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:52.670502    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:52.681269 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:52.681250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:53.487910 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:53.487880    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:53.499580 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:53.499558    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:53.508786 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:53.508750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:53.520193 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:53.520173    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:54.319673 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:54.319645    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:54.332589 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:54.332563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:54.343409 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:54.343383    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:54.356107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:54.356085    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:55.171936 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:55.171862    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:55.206566 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:55.206541    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:55.243872 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:55.243851    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:55.272305 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:55.272285    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:56.116172 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:56.116142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:56.130712 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:56.130683    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:56.142857 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:56.142834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:56.154897 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:56.154871    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:57.235221 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:57.235177    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:57.247168 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:57.247138    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:57.256325 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:57.256298    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:57.267148 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:57.267120    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:57.432373 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:57.432339    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 18 20:46:58.140663 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:58.140627    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:58.154083 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:58.154056    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:58.166791 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:58.166769    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:58.178245 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:58.178222    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:59.036069 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.036042    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kserve-container/0.log"
Sep 18 20:46:59.048753 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.048726    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/kube-rbac-proxy/0.log"
Sep 18 20:46:59.060780 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.060759    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/agent/0.log"
Sep 18 20:46:59.072082 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.072062    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-h5cf7_19b1feec-b214-4317-b226-fe41a312b38c/storage-initializer/0.log"
Sep 18 20:46:59.835408 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.835384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:46:59.864816 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.864795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-cabundle-cert\") pod \"19b1feec-b214-4317-b226-fe41a312b38c\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") "
Sep 18 20:46:59.864963 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.864875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls\") pod \"19b1feec-b214-4317-b226-fe41a312b38c\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") "
Sep 18 20:46:59.864963 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.864895    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b1feec-b214-4317-b226-fe41a312b38c-kserve-provision-location\") pod \"19b1feec-b214-4317-b226-fe41a312b38c\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") "
Sep 18 20:46:59.864963 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.864924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98d9c\" (UniqueName: \"kubernetes.io/projected/19b1feec-b214-4317-b226-fe41a312b38c-kube-api-access-98d9c\") pod \"19b1feec-b214-4317-b226-fe41a312b38c\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") "
Sep 18 20:46:59.864963 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.864942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-kube-rbac-proxy-sar-config\") pod \"19b1feec-b214-4317-b226-fe41a312b38c\" (UID: \"19b1feec-b214-4317-b226-fe41a312b38c\") "
Sep 18 20:46:59.865244 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.865209    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19b1feec-b214-4317-b226-fe41a312b38c" (UID: "19b1feec-b214-4317-b226-fe41a312b38c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:46:59.865244 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.865222    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b1feec-b214-4317-b226-fe41a312b38c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19b1feec-b214-4317-b226-fe41a312b38c" (UID: "19b1feec-b214-4317-b226-fe41a312b38c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:46:59.865367 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.865340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19b1feec-b214-4317-b226-fe41a312b38c" (UID: "19b1feec-b214-4317-b226-fe41a312b38c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:46:59.866971 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.866945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b1feec-b214-4317-b226-fe41a312b38c-kube-api-access-98d9c" (OuterVolumeSpecName: "kube-api-access-98d9c") pod "19b1feec-b214-4317-b226-fe41a312b38c" (UID: "19b1feec-b214-4317-b226-fe41a312b38c"). InnerVolumeSpecName "kube-api-access-98d9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:46:59.867088 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.867017    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19b1feec-b214-4317-b226-fe41a312b38c" (UID: "19b1feec-b214-4317-b226-fe41a312b38c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:46:59.912334 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.912310    2577 generic.go:356] "Generic (PLEG): container finished" podID="19b1feec-b214-4317-b226-fe41a312b38c" containerID="24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc" exitCode=0
Sep 18 20:46:59.912427 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.912367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerDied","Data":"24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc"}
Sep 18 20:46:59.912427 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.912381    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"
Sep 18 20:46:59.912570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.912434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7" event={"ID":"19b1feec-b214-4317-b226-fe41a312b38c","Type":"ContainerDied","Data":"0acb5bf5d094a39548d6a1651e0cecb080e7d043f5f6ef7e2b11eae6ca5125d1"}
Sep 18 20:46:59.912570 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.912466    2577 scope.go:117] "RemoveContainer" containerID="24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc"
Sep 18 20:46:59.914333 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.914313    2577 generic.go:356] "Generic (PLEG): container finished" podID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerID="ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b" exitCode=0
Sep 18 20:46:59.914421 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.914382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qczb/must-gather-fh7b9" event={"ID":"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b","Type":"ContainerDied","Data":"ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b"}
Sep 18 20:46:59.914693 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.914679    2577 scope.go:117] "RemoveContainer" containerID="ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b"
Sep 18 20:46:59.920495 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.920461    2577 scope.go:117] "RemoveContainer" containerID="29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920"
Sep 18 20:46:59.927293 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.927277    2577 scope.go:117] "RemoveContainer" containerID="f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8"
Sep 18 20:46:59.934226 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.934211    2577 scope.go:117] "RemoveContainer" containerID="e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff"
Sep 18 20:46:59.940790 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.940775    2577 scope.go:117] "RemoveContainer" containerID="24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc"
Sep 18 20:46:59.941057 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:46:59.941040    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc\": container with ID starting with 24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc not found: ID does not exist" containerID="24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc"
Sep 18 20:46:59.941140 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.941066    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc"} err="failed to get container status \"24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc\": rpc error: code = NotFound desc = could not find container \"24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc\": container with ID starting with 24c9114c7c50e651cd2d312b839c81beffc7b4e220230a9334f28c6f1f022cdc not found: ID does not exist"
Sep 18 20:46:59.941140 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.941089    2577 scope.go:117] "RemoveContainer" containerID="29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920"
Sep 18 20:46:59.941343 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:46:59.941327    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920\": container with ID starting with 29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920 not found: ID does not exist" containerID="29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920"
Sep 18 20:46:59.941383 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.941350    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920"} err="failed to get container status \"29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920\": rpc error: code = NotFound desc = could not find container \"29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920\": container with ID starting with 29b017df51cfcae09cba3f58c5bcb57b78a79783b902f8a2310a93fd9b9e2920 not found: ID does not exist"
Sep 18 20:46:59.941383 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.941364    2577 scope.go:117] "RemoveContainer" containerID="f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8"
Sep 18 20:46:59.941633 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:46:59.941616    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8\": container with ID starting with f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8 not found: ID does not exist" containerID="f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8"
Sep 18 20:46:59.941682 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.941638    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8"} err="failed to get container status \"f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8\": rpc error: code = NotFound desc = could not find container \"f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8\": container with ID starting with f7dead7ca616a7f9edd251a24bd855ff0e9d816f619eea98aacfdb803adf68f8 not found: ID does not exist"
Sep 18 20:46:59.941682 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.941652    2577 scope.go:117] "RemoveContainer" containerID="e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff"
Sep 18 20:46:59.941871 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:46:59.941855    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff\": container with ID starting with e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff not found: ID does not exist" containerID="e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff"
Sep 18 20:46:59.941930 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.941881    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff"} err="failed to get container status \"e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff\": rpc error: code = NotFound desc = could not find container \"e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff\": container with ID starting with e09e7d9ff9e6e7dfbbacc1437fa1352059b7083bb9b7090e2fe6f0459ebfd8ff not found: ID does not exist"
Sep 18 20:46:59.944001 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.943981    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"]
Sep 18 20:46:59.949696 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.949650    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-h5cf7"]
Sep 18 20:46:59.966028 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.966013    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b1feec-b214-4317-b226-fe41a312b38c-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:46:59.966107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.966030    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b1feec-b214-4317-b226-fe41a312b38c-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:46:59.966107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.966040    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98d9c\" (UniqueName: \"kubernetes.io/projected/19b1feec-b214-4317-b226-fe41a312b38c-kube-api-access-98d9c\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:46:59.966107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.966049    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:46:59.966107 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:46:59.966058    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b1feec-b214-4317-b226-fe41a312b38c-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:47:00.726073 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:00.726040    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4qczb_must-gather-fh7b9_688dbfb2-d3b6-4c79-8fb7-51f05569ed9b/gather/0.log"
Sep 18 20:47:01.243016 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.242984    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19b1feec-b214-4317-b226-fe41a312b38c" path="/var/lib/kubelet/pods/19b1feec-b214-4317-b226-fe41a312b38c/volumes"
Sep 18 20:47:01.285409 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285381    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dcbdm/must-gather-h487q"]
Sep 18 20:47:01.285686 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285674    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="storage-initializer"
Sep 18 20:47:01.285686 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285687    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="storage-initializer"
Sep 18 20:47:01.285777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285701    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent"
Sep 18 20:47:01.285777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285707    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent"
Sep 18 20:47:01.285777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285716    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container"
Sep 18 20:47:01.285777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285722    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container"
Sep 18 20:47:01.285777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285728    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy"
Sep 18 20:47:01.285777 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285734    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy"
Sep 18 20:47:01.285984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285782    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="agent"
Sep 18 20:47:01.285984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285789    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kserve-container"
Sep 18 20:47:01.285984 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.285795    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19b1feec-b214-4317-b226-fe41a312b38c" containerName="kube-rbac-proxy"
Sep 18 20:47:01.289804 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.289788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcbdm/must-gather-h487q"
Sep 18 20:47:01.292274 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.292249    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dcbdm\"/\"default-dockercfg-l6fdb\""
Sep 18 20:47:01.292274 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.292270    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dcbdm\"/\"kube-root-ca.crt\""
Sep 18 20:47:01.292422 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.292349    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dcbdm\"/\"openshift-service-ca.crt\""
Sep 18 20:47:01.298438 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.298415    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcbdm/must-gather-h487q"]
Sep 18 20:47:01.376295 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.376269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54k2\" (UniqueName: \"kubernetes.io/projected/f6ed234b-356d-4829-9b6c-8363ec312716-kube-api-access-s54k2\") pod \"must-gather-h487q\" (UID: \"f6ed234b-356d-4829-9b6c-8363ec312716\") " pod="openshift-must-gather-dcbdm/must-gather-h487q"
Sep 18 20:47:01.376416 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.376319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f6ed234b-356d-4829-9b6c-8363ec312716-must-gather-output\") pod \"must-gather-h487q\" (UID: \"f6ed234b-356d-4829-9b6c-8363ec312716\") " pod="openshift-must-gather-dcbdm/must-gather-h487q"
Sep 18 20:47:01.477196 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.477173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54k2\" (UniqueName: \"kubernetes.io/projected/f6ed234b-356d-4829-9b6c-8363ec312716-kube-api-access-s54k2\") pod \"must-gather-h487q\" (UID: \"f6ed234b-356d-4829-9b6c-8363ec312716\") " pod="openshift-must-gather-dcbdm/must-gather-h487q"
Sep 18 20:47:01.477319 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.477219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f6ed234b-356d-4829-9b6c-8363ec312716-must-gather-output\") pod \"must-gather-h487q\" (UID: \"f6ed234b-356d-4829-9b6c-8363ec312716\") " pod="openshift-must-gather-dcbdm/must-gather-h487q"
Sep 18 20:47:01.477522 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.477505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f6ed234b-356d-4829-9b6c-8363ec312716-must-gather-output\") pod \"must-gather-h487q\" (UID: \"f6ed234b-356d-4829-9b6c-8363ec312716\") " pod="openshift-must-gather-dcbdm/must-gather-h487q"
Sep 18 20:47:01.485613 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.485595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54k2\" (UniqueName: \"kubernetes.io/projected/f6ed234b-356d-4829-9b6c-8363ec312716-kube-api-access-s54k2\") pod \"must-gather-h487q\" (UID: \"f6ed234b-356d-4829-9b6c-8363ec312716\") " pod="openshift-must-gather-dcbdm/must-gather-h487q"
Sep 18 20:47:01.599689 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.599667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcbdm/must-gather-h487q"
Sep 18 20:47:01.869276 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.869210    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcbdm/must-gather-h487q"]
Sep 18 20:47:01.871332 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:47:01.871309    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6ed234b_356d_4829_9b6c_8363ec312716.slice/crio-e919309007d5f023cfb68d43feae86ca474d0ab3d099389881fa4eb2e7b25fef WatchSource:0}: Error finding container e919309007d5f023cfb68d43feae86ca474d0ab3d099389881fa4eb2e7b25fef: Status 404 returned error can't find the container with id e919309007d5f023cfb68d43feae86ca474d0ab3d099389881fa4eb2e7b25fef
Sep 18 20:47:01.923356 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:01.923327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcbdm/must-gather-h487q" event={"ID":"f6ed234b-356d-4829-9b6c-8363ec312716","Type":"ContainerStarted","Data":"e919309007d5f023cfb68d43feae86ca474d0ab3d099389881fa4eb2e7b25fef"}
Sep 18 20:47:02.930230 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:02.930201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcbdm/must-gather-h487q" event={"ID":"f6ed234b-356d-4829-9b6c-8363ec312716","Type":"ContainerStarted","Data":"7139ed293a8996bb0e231b75b3e92781beb302b85c5eb2d1f76b4f49121d1b81"}
Sep 18 20:47:03.936180 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:03.936143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcbdm/must-gather-h487q" event={"ID":"f6ed234b-356d-4829-9b6c-8363ec312716","Type":"ContainerStarted","Data":"48c6eba53278b9c8671db5e62a819a07c594159577ea92145268bcbff1f72bcc"}
Sep 18 20:47:03.953314 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:03.953246    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dcbdm/must-gather-h487q" podStartSLOduration=2.054702678 podStartE2EDuration="2.953223072s" totalImagesPullingTime="898.520394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:47:03.951554102 +0000 UTC m=+1600.761340044" watchObservedRunningTime="2026-09-18 20:47:03.953223072 +0000 UTC m=+1600.763009016"
Sep 18 20:47:04.166744 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:04.166716    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b8ng8_27bf6b08-ab66-451f-bc9b-ce188564686c/global-pull-secret-syncer/0.log"
Sep 18 20:47:04.339053 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:04.339026    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g44d8_296cdbc3-c56a-422c-bc96-74c175ddfcc6/konnectivity-agent/0.log"
Sep 18 20:47:04.412075 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:04.412046    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-158.ec2.internal_9544b8965977bc8ce7945a3e09bc38b4/haproxy/0.log"
Sep 18 20:47:06.161306 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.161247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-4qczb/must-gather-fh7b9"]
Sep 18 20:47:06.164325 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.164300    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-4qczb/must-gather-fh7b9"]
Sep 18 20:47:06.164992 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.164684    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-4qczb/must-gather-fh7b9" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerName="copy" containerID="cri-o://cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43" gracePeriod=2
Sep 18 20:47:06.168359 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:47:06.168310    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fh7b9\" is forbidden: User \"system:node:ip-10-0-138-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4qczb\": no relationship found between node 'ip-10-0-138-158.ec2.internal' and this object" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:47:06.575505 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.572752    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4qczb_must-gather-fh7b9_688dbfb2-d3b6-4c79-8fb7-51f05569ed9b/copy/0.log"
Sep 18 20:47:06.581492 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.578542    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:47:06.581492 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:47:06.580901    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fh7b9\" is forbidden: User \"system:node:ip-10-0-138-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4qczb\": no relationship found between node 'ip-10-0-138-158.ec2.internal' and this object" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:47:06.621499 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.618509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-868vk\" (UniqueName: \"kubernetes.io/projected/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-kube-api-access-868vk\") pod \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\" (UID: \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\") "
Sep 18 20:47:06.621499 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.620703    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" (UID: "688dbfb2-d3b6-4c79-8fb7-51f05569ed9b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:06.621499 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.620780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-must-gather-output\") pod \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\" (UID: \"688dbfb2-d3b6-4c79-8fb7-51f05569ed9b\") "
Sep 18 20:47:06.623759 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.623342    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-must-gather-output\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:47:06.635499 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.633546    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-kube-api-access-868vk" (OuterVolumeSpecName: "kube-api-access-868vk") pod "688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" (UID: "688dbfb2-d3b6-4c79-8fb7-51f05569ed9b"). InnerVolumeSpecName "kube-api-access-868vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:47:06.724279 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.724235    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-868vk\" (UniqueName: \"kubernetes.io/projected/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b-kube-api-access-868vk\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Sep 18 20:47:06.975398 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.975355    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4qczb_must-gather-fh7b9_688dbfb2-d3b6-4c79-8fb7-51f05569ed9b/copy/0.log"
Sep 18 20:47:06.975875 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.975845    2577 generic.go:356] "Generic (PLEG): container finished" podID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerID="cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43" exitCode=143
Sep 18 20:47:06.975995 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.975968    2577 scope.go:117] "RemoveContainer" containerID="cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43"
Sep 18 20:47:06.976132 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.976118    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:47:06.980011 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:47:06.979979    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fh7b9\" is forbidden: User \"system:node:ip-10-0-138-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4qczb\": no relationship found between node 'ip-10-0-138-158.ec2.internal' and this object" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:47:06.996966 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:06.996945    2577 scope.go:117] "RemoveContainer" containerID="ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b"
Sep 18 20:47:06.999507 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:47:06.999460    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fh7b9\" is forbidden: User \"system:node:ip-10-0-138-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4qczb\": no relationship found between node 'ip-10-0-138-158.ec2.internal' and this object" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" pod="openshift-must-gather-4qczb/must-gather-fh7b9"
Sep 18 20:47:07.017769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:07.015595    2577 scope.go:117] "RemoveContainer" containerID="cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43"
Sep 18 20:47:07.017769 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:47:07.015999    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43\": container with ID starting with cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43 not found: ID does not exist" containerID="cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43"
Sep 18 20:47:07.017769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:07.016047    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43"} err="failed to get container status \"cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43\": rpc error: code = NotFound desc = could not find container \"cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43\": container with ID starting with cdd350104ce44b23d383c4767fcfb8d01673afd8bbdf4533abf2ee7593f23f43 not found: ID does not exist"
Sep 18 20:47:07.017769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:07.016072    2577 scope.go:117] "RemoveContainer" containerID="ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b"
Sep 18 20:47:07.017769 ip-10-0-138-158 kubenswrapper[2577]: E0918 20:47:07.016373    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b\": container with ID starting with ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b not found: ID does not exist" containerID="ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b"
Sep 18 20:47:07.017769 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:07.016396    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b"} err="failed to get container status \"ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b\": rpc error: code = NotFound desc = could not find container \"ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b\": container with ID starting with ba4291d4a041fb90e6667583f2fafdfe1aae87c9c8bd862f164a22c4d067b52b not found: ID does not exist"
Sep 18 20:47:07.252250 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:07.252133    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" path="/var/lib/kubelet/pods/688dbfb2-d3b6-4c79-8fb7-51f05569ed9b/volumes"
Sep 18 20:47:08.960041 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:08.959970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h8hp2_eedd0741-cae1-4c0f-8bb6-616b92088ca6/node-exporter/0.log"
Sep 18 20:47:08.982781 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:08.982745    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h8hp2_eedd0741-cae1-4c0f-8bb6-616b92088ca6/kube-rbac-proxy/0.log"
Sep 18 20:47:09.006838 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.006803    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h8hp2_eedd0741-cae1-4c0f-8bb6-616b92088ca6/init-textfile/0.log"
Sep 18 20:47:09.216628 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.216553    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0786959-e2d1-4d36-8517-dc2ccb452333/prometheus/0.log"
Sep 18 20:47:09.242904 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.242522    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0786959-e2d1-4d36-8517-dc2ccb452333/config-reloader/0.log"
Sep 18 20:47:09.268379 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.268336    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0786959-e2d1-4d36-8517-dc2ccb452333/thanos-sidecar/0.log"
Sep 18 20:47:09.302396 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.302362    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0786959-e2d1-4d36-8517-dc2ccb452333/kube-rbac-proxy-web/0.log"
Sep 18 20:47:09.329390 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.329358    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0786959-e2d1-4d36-8517-dc2ccb452333/kube-rbac-proxy/0.log"
Sep 18 20:47:09.354896 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.354865    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0786959-e2d1-4d36-8517-dc2ccb452333/kube-rbac-proxy-thanos/0.log"
Sep 18 20:47:09.385360 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.385326    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0786959-e2d1-4d36-8517-dc2ccb452333/init-config-reloader/0.log"
Sep 18 20:47:09.517235 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.517154    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79959c8594-km6fw_87374a97-791b-47b8-a9db-748a6504be64/telemeter-client/0.log"
Sep 18 20:47:09.549404 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.549372    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79959c8594-km6fw_87374a97-791b-47b8-a9db-748a6504be64/reload/0.log"
Sep 18 20:47:09.575374 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.574250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79959c8594-km6fw_87374a97-791b-47b8-a9db-748a6504be64/kube-rbac-proxy/0.log"
Sep 18 20:47:09.609929 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.609879    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76459647d6-gxxbg_deb1487d-e41c-4a40-bba4-12349dcc1741/thanos-query/0.log"
Sep 18 20:47:09.638758 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.638716    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76459647d6-gxxbg_deb1487d-e41c-4a40-bba4-12349dcc1741/kube-rbac-proxy-web/0.log"
Sep 18 20:47:09.666325 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.666293    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76459647d6-gxxbg_deb1487d-e41c-4a40-bba4-12349dcc1741/kube-rbac-proxy/0.log"
Sep 18 20:47:09.689636 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.689607    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76459647d6-gxxbg_deb1487d-e41c-4a40-bba4-12349dcc1741/prom-label-proxy/0.log"
Sep 18 20:47:09.717240 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.717191    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76459647d6-gxxbg_deb1487d-e41c-4a40-bba4-12349dcc1741/kube-rbac-proxy-rules/0.log"
Sep 18 20:47:09.743289 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:09.743261    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76459647d6-gxxbg_deb1487d-e41c-4a40-bba4-12349dcc1741/kube-rbac-proxy-metrics/0.log"
Sep 18 20:47:12.203054 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.203026    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"]
Sep 18 20:47:12.203523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.203299    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerName="gather"
Sep 18 20:47:12.203523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.203310    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerName="gather"
Sep 18 20:47:12.203523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.203323    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerName="copy"
Sep 18 20:47:12.203523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.203328    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerName="copy"
Sep 18 20:47:12.203523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.203377    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerName="gather"
Sep 18 20:47:12.203523 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.203383    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="688dbfb2-d3b6-4c79-8fb7-51f05569ed9b" containerName="copy"
Sep 18 20:47:12.206435 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.206403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.217414 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.217383    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"]
Sep 18 20:47:12.275691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.274630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-sys\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.275691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.274769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-proc\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.275691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.274875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-podres\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.275691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.274900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssjc\" (UniqueName: \"kubernetes.io/projected/2758c566-fa17-466a-b6ce-39ee23a37918-kube-api-access-bssjc\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.275691 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.274935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-lib-modules\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.376378 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.376324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-podres\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.376378 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.376379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssjc\" (UniqueName: \"kubernetes.io/projected/2758c566-fa17-466a-b6ce-39ee23a37918-kube-api-access-bssjc\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.376662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.376416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-lib-modules\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.376662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.376467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-sys\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.376662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.376527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-proc\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.376662 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.376625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-proc\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.376893 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.376748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-podres\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.377071 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.377043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-sys\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.377204 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.377105    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2758c566-fa17-466a-b6ce-39ee23a37918-lib-modules\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.385101 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.385066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssjc\" (UniqueName: \"kubernetes.io/projected/2758c566-fa17-466a-b6ce-39ee23a37918-kube-api-access-bssjc\") pod \"perf-node-gather-daemonset-8pcpd\" (UID: \"2758c566-fa17-466a-b6ce-39ee23a37918\") " pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.518211 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.518139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:12.808825 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:12.808747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"]
Sep 18 20:47:12.814031 ip-10-0-138-158 kubenswrapper[2577]: W0918 20:47:12.814000    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2758c566_fa17_466a_b6ce_39ee23a37918.slice/crio-4f22deb7fa6c192fd7473ce3f634acfaea2078340a351cf5a609092235eda4d3 WatchSource:0}: Error finding container 4f22deb7fa6c192fd7473ce3f634acfaea2078340a351cf5a609092235eda4d3: Status 404 returned error can't find the container with id 4f22deb7fa6c192fd7473ce3f634acfaea2078340a351cf5a609092235eda4d3
Sep 18 20:47:13.012379 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:13.012343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd" event={"ID":"2758c566-fa17-466a-b6ce-39ee23a37918","Type":"ContainerStarted","Data":"c0e1aafb379171ae914bdf18cd35eee39acc1119cd8fc6a1b70bf4eb81f15cf1"}
Sep 18 20:47:13.012575 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:13.012408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd" event={"ID":"2758c566-fa17-466a-b6ce-39ee23a37918","Type":"ContainerStarted","Data":"4f22deb7fa6c192fd7473ce3f634acfaea2078340a351cf5a609092235eda4d3"}
Sep 18 20:47:13.012575 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:13.012439    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:13.028441 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:13.028381    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd" podStartSLOduration=1.028362842 podStartE2EDuration="1.028362842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:47:13.027870649 +0000 UTC m=+1609.837656594" watchObservedRunningTime="2026-09-18 20:47:13.028362842 +0000 UTC m=+1609.838148785"
Sep 18 20:47:13.562189 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:13.562161    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7xtmj_79e14693-08d8-4f26-bc83-a0ca0dcd8073/dns/0.log"
Sep 18 20:47:13.589367 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:13.589337    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7xtmj_79e14693-08d8-4f26-bc83-a0ca0dcd8073/kube-rbac-proxy/0.log"
Sep 18 20:47:13.661394 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:13.661371    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kpgfw_244a261e-ab8c-41a4-9a04-09b22da8a488/dns-node-resolver/0.log"
Sep 18 20:47:14.178307 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:14.178280    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gzrh4_7fa951d2-1593-4871-b3b3-76614e660378/node-ca/0.log"
Sep 18 20:47:15.297036 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:15.297010    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l6gq4_5c36bc1f-ebd4-4c94-aa48-51ed310e349f/serve-healthcheck-canary/0.log"
Sep 18 20:47:15.926335 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:15.926306    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ltk7w_385a1e4e-2987-4cfd-acf7-69508486b476/kube-rbac-proxy/0.log"
Sep 18 20:47:15.953087 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:15.953060    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ltk7w_385a1e4e-2987-4cfd-acf7-69508486b476/exporter/0.log"
Sep 18 20:47:15.980310 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:15.980276    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ltk7w_385a1e4e-2987-4cfd-acf7-69508486b476/extractor/0.log"
Sep 18 20:47:18.317204 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:18.317174    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-77rbj_edc1147e-f3cf-4387-a4ec-3ac1e529f050/manager/0.log"
Sep 18 20:47:18.504557 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:18.504527    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bp7r8_d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8/s3-init/0.log"
Sep 18 20:47:18.522917 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:18.522892    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bp7r8_d7ff47d2-eeec-4b7a-a6f4-06693d2c58c8/download-hf-model/0.log"
Sep 18 20:47:19.029262 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:19.029227    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dcbdm/perf-node-gather-daemonset-8pcpd"
Sep 18 20:47:22.907211 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:22.907180    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-wwknc_a0e95428-8170-4f7e-b84c-26bcf40dbd69/migrator/0.log"
Sep 18 20:47:22.933387 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:22.933361    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-wwknc_a0e95428-8170-4f7e-b84c-26bcf40dbd69/graceful-termination/0.log"
Sep 18 20:47:24.335000 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.334977    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2nh2_d77a339a-d325-4683-a6ae-945a5b0ebae1/kube-multus-additional-cni-plugins/0.log"
Sep 18 20:47:24.360400 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.360375    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2nh2_d77a339a-d325-4683-a6ae-945a5b0ebae1/egress-router-binary-copy/0.log"
Sep 18 20:47:24.382398 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.382380    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2nh2_d77a339a-d325-4683-a6ae-945a5b0ebae1/cni-plugins/0.log"
Sep 18 20:47:24.404294 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.404277    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2nh2_d77a339a-d325-4683-a6ae-945a5b0ebae1/bond-cni-plugin/0.log"
Sep 18 20:47:24.427586 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.427566    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2nh2_d77a339a-d325-4683-a6ae-945a5b0ebae1/routeoverride-cni/0.log"
Sep 18 20:47:24.455949 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.455923    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2nh2_d77a339a-d325-4683-a6ae-945a5b0ebae1/whereabouts-cni-bincopy/0.log"
Sep 18 20:47:24.480042 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.480022    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2nh2_d77a339a-d325-4683-a6ae-945a5b0ebae1/whereabouts-cni/0.log"
Sep 18 20:47:24.734809 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.734737    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c9w4s_22f68618-1475-4194-99f3-cb41d4662db9/kube-multus/0.log"
Sep 18 20:47:24.907833 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.907808    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rhttn_7795f7f8-da44-460a-a884-07d834c38ba3/network-metrics-daemon/0.log"
Sep 18 20:47:24.969262 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:24.969234    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rhttn_7795f7f8-da44-460a-a884-07d834c38ba3/kube-rbac-proxy/0.log"
Sep 18 20:47:25.813461 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:25.813370    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dklfc_009a3dff-41dc-4ee6-9d76-d36b1572b38c/ovn-controller/0.log"
Sep 18 20:47:25.834185 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:25.834163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dklfc_009a3dff-41dc-4ee6-9d76-d36b1572b38c/ovn-acl-logging/0.log"
Sep 18 20:47:25.860796 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:25.860771    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dklfc_009a3dff-41dc-4ee6-9d76-d36b1572b38c/kube-rbac-proxy-node/0.log"
Sep 18 20:47:25.882785 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:25.882763    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dklfc_009a3dff-41dc-4ee6-9d76-d36b1572b38c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 20:47:25.904036 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:25.904018    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dklfc_009a3dff-41dc-4ee6-9d76-d36b1572b38c/northd/0.log"
Sep 18 20:47:25.929981 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:25.929965    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dklfc_009a3dff-41dc-4ee6-9d76-d36b1572b38c/nbdb/0.log"
Sep 18 20:47:25.957092 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:25.957075    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dklfc_009a3dff-41dc-4ee6-9d76-d36b1572b38c/sbdb/0.log"
Sep 18 20:47:26.023099 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:26.023064    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dklfc_009a3dff-41dc-4ee6-9d76-d36b1572b38c/ovnkube-controller/0.log"
Sep 18 20:47:27.551024 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:27.550996    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-hptqd_fa298501-12e5-49d3-9272-b6a6f045d362/network-check-target-container/0.log"
Sep 18 20:47:28.583922 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:28.583892    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bsrdj_d9688a58-6f94-40c5-bb65-85b0fd7bf192/iptables-alerter/0.log"
Sep 18 20:47:29.385900 ip-10-0-138-158 kubenswrapper[2577]: I0918 20:47:29.385849    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-q7jwp_f5c64d04-78fe-440d-81d1-b17abb3c02e0/tuned/0.log"